12/17/2023 0 Comments Googgle documentsWhether the same thing will happen for LLMs remains to be seen, but the broad structural elements are the same. The effect was palpable: rapid domination in terms of cultural impact vs the OpenAI solution, which became increasingly irrelevant. Having an open model led to product integrations, marketplaces, user interfaces, and innovations that didn’t happen for Dall-E. These contributions were pivotal in the image generation space, setting Stable Diffusion on a different path from Dall-E. In both cases, this quickly outpaced the large players. In both cases, access to a sufficiently high-quality model kicked off a flurry of ideas and iteration from individuals and institutions around the world. In both cases, low-cost public involvement was enabled by a vastly cheaper mechanism for fine tuning called low rank adaptation, or LoRA, combined with a significant breakthrough in scale ( latent diffusion for image synthesis, Chinchilla for LLMs). The similarities are not lost on the community, with many calling this the “ Stable Diffusion moment ” for LLMs. The current renaissance in open source LLMs comes hot on the heels of a renaissance in image generation. In many ways, this shouldn’t be a surprise to anyone. We should make small variants more than an afterthought, now that we know what is possible in the <20B parameter regime. In the long run, the best models are the ones We should consider where our value add really is. People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. We should prioritize enabling 3P integrations. Our best hope is to learn from and collaborate with what others are doing outside Google. And they are doing so in weeks, not months. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Multimodality: The current multimodal ScienceQA SOTA was trained in an hour. There are entire websites full of art models with no restrictions whatsoever, and text is not far behind. Responsible Release: This one isn’t “solved” so much as “obviated”. Scalable Personal AI: You can finetune a personalized AI on your laptop in an evening. LLMs on a Phone: People are running foundation models on a Pixel 6 at 5 tokens / sec. Things we consider “major open problems” are solved and in people’s hands today. I’m talking, of course, about open source. While we’ve been squabbling, a third faction has been quietly eating our lunch. Who will cross the next milestone? What will the next move be?īut the uncomfortable truth is, we aren’t positioned to win this arms race and neither is OpenAI. We’ve done a lot of looking over our shoulders at OpenAI.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |