I’ll often ask GPT to explain a concept to me, but giving it a hint as to my existing level of knowledge in the subject area. With Google, you’re likely to get the most popular page on a topic; not the page that is appropriate for people with your expertise.
For instance, if I want to know about extreme ultraviolet lithography and I tell GPT-4 that I have a degree in engineering and that I studied advanced optics, the explanation is much richer in useful detail, going way beyond what any of the pages I could find on Google would reveal.
Hmm, I tried some questions which would have been relevant to me in the recent past, and it flubbed pretty hard on all of them. Even worse, instead of just saying it doesn't know it generates semi-plausible babbling or adjacent but not actually helpful knowledge.
I decided to give it another go and ask GPT-4 three questions which I needed to get answers to within the last few months.
Asking conceptually about DPO: https://chat.openai.com/share/6611454c-60de-4317-811b-2b7f31... - In this one it completely leaves out the actual trick which enables DPO, so I would say it has almost no information content. Someone who didn't know what DPO is and read this would incorrectly think that they had learned something. - To learn about this, the right place was to read the original DPO paper, and some follow-up work
Asking about FSDP compatibility with LoRA: https://chat.openai.com/share/5f8892ea-61e6-496f-abda-d5a8ad... - In this one it just says a bunch of generic vague things without answering the question. - The right place to learn the answer to this is diving through Github issue comments
Asking for details the MegaBlocks mixture-of-experts setup: https://chat.openai.com/share/c010e630-ba08-407e-afb3-03df99... - Again it's just saying generic stuff which is relevant to mixture-of-experts in general, but it leaves out everything that actually makes the MegaBlocks MoE different from a generic MoE idea - For this one I had to do a combination of reading the paper and the MegaBlocks repo
So 0/3 and pretty dramatically. I was actually expecting it to get at least one of those. As far as I can tell, it didn't really do anything different based on me specifying my background either. I'd love to see any links to productive conversations that people can share.
That’s an area where they genuinely suck, unless there is an exact wikipedia article on the topic, readily within its training data set.
If you wander off of that knowledge sphere, and you are not sufficiently knowledgeable yourself about the topic, it can tell you some really stupid stuff.
Nonetheless, I do use it quite regularly in everyday life, as it is basically the best reverse dictionary (that works for any language) there is. For work (programming), I didn’t find a better use case than sometimes passing it a list of stuff, giving it an example on the first element on what I want, and using it to generate it for the rest of the elements.
But that is <1% of what I do each day.
Reverse dictionary is a great one yeah. I think I've used it for resolving a tip-of-my tongue thing several times. For language learning I'm a bit more wary. A few times I've asked it for help with Chinese, and when I ask one of my native speaker friends they'll tell me it wasn't quite right.