Mapping the future of *truly* Open Models and Training Dolly for $30 — with Mike Conover of Databricks
The race is on for the first fully GPT3/4-equivalent, truly open source Foundation Model! LLaMA’s release proved that a great model could be released and run on consumer-grade hardware (see llama.cpp), but its research license prohibits businesses from running it and all it’s variants (Alpaca, Vicuna, Koala, etc) for their own use at work. So there is great interest and desire for *truly* open source LLMs that are feasible for commercial use (with far better customization, finetuning, and privacy than the closed source LLM APIs).
The previous leading contenders were Eleuther’s GPT-J and Neo on the small end (Mike Conover tells the story of how a small team of Applied AI engineers got convinced Ali Ghodsi and 5,000 of their coworkers to join in the adventure of building the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use. He also indulges our questions on other recent open source LLM projects, CerebasGPT and RedPajama, though we recorded this a week before Stability’s StableLM release.
Stick around to the end for some easter eggs featuring AI Drake!
Recorded in-person at the beautiful StudioPod studios in San Francisco.
Full transcript is below the fold.
Show Notes
* Mike Conover LinkedInand Twitter
* Dolly 1.0
* Dolly 2.0
* CICERO and Diplomacy
* Dolly and Deepspeed
* LLMops:
* https://nat.dev/
* PromptLayer
* HumanLoop
* Spreadsheets??
* Quadratic
* Alessio’s Email GPT Drafter
* Open Models
* Open Assistant
* Cerebras GPT
* RedPajama
* Reflexion, Recursive Criticism and Improvement
* Lightning Round
* AI Product: Google Maps
* AI People: EleutherAI, Huggingface’s Stas Bekman
* AI Prediction: Open LLaMA reproduction, AI Twins of People (AI Drake), Valuing Perplexity
* Request for Startups: LLMOps/Benchmarks, Trail Mapping
Timestamps
* [00:00:21] Introducing Mike Conover
* [00:03:10] Dolly 1.0
* [00:04:18] Making Dolly
* [00:06:12] Dolly 2.0
* [00:09:28] Gamifying Instruction Tuning
* [00:11:36] Summarization - Thumbnails for Language
* [00:15:11] CICERO and Geopolitical AI Agents
* [00:17:09] Datasets vs Intentional Design
* [00:21:44] Biological Basis of AI
* [00:23:27] Training Your Own LLMs
* [00:28:21] You May Not Need a Large Model
* [00:29:59] Good LLM Use cases
* [00:31:33] Dolly Cost $30 on Databricks
* [00:36:06] Databricks Open Source
* [00:37:31] LLMOps and Prompt Tooling
* [00:42:26] "I'm a Sheets Maxi"
* [00:44:19] AI and Workplace Productivity
* [00:47:02] OpenAssistant
* [00:47:41] CerebrasGPT
* [00:51:35] RedPajama
* [00:54:07] Why Dolly > OpenAI GPT
* [00:56:19] Open Source Licensing for AI Models
* [00:57:09] Why O
