# Research Base 1. [LLMs are Not Creative](LLMs%20are%20Not%20Creative.md) 2. [LLMs are Creative](LLMs%20are%20Creative.md) # Queue ### Queue (Specific Arguments to grab) 1. Pull together David Deutsch Resources on AGI and Creativity 1. what does David deutsch mean when he says "universal explainer"? How does this relate to universal computer? 2. Brett hall inside of sun example from reason is fun podcast 2. knowledge creation from popper (see the book philosophy in the real world) 3. [#61: Prof. YANN LECUN: Interpolation, Extrapolation and Linearisation (w/ Dr. Randall Balestriero) - YouTube](https://www.youtube.com/watch?v=86ib0sfdFtw&t=1958s) 4. [What to Expect When You’re Expecting … GPT-4](https://garymarcus.substack.com/p/what-to-expect-when-youre-expecting) ### Queue (Directly Related) 1. [https://www.daviddeutsch.org.uk/wp-content/uploads/2019/07/PossibleMinds_Deutsch.pdf](https://www.daviddeutsch.org.uk/wp-content/uploads/2019/07/PossibleMinds_Deutsch.pdf) 2. [https://aeon.co/essays/how-close-are-we-to-creating-artificial-intelligence](https://aeon.co/essays/how-close-are-we-to-creating-artificial-intelligence) 3. General Ideas about AI in the world today 1. [https://a16z.com/2023/06/06/ai-will-save-the-world/](https://a16z.com/2023/06/06/ai-will-save-the-world/) 2. [Why transformative artificial intelligence is really, really hard to achieve](https://thegradient.pub/why-transformative-artificial-intelligence-is-really-really-hard-to-achieve/) 3. [What if we could automate invention? - by Matt Clancy](https://mattsclancy.substack.com/p/what-if-we-could-automate-invention) (listen to podcast version) 4. Natural General Intelligence Book 5. Reason is fun, first episode w/ David Deutsch 6. On Measures of Intelligence Francois Chollet 7. Kenneth Stanley 1. [#72 Prof. KEN STANLEY 2.0 - On Art and Subjectivity [UNPLUGGED] - YouTube](https://www.youtube.com/watch?v=DxBZORM9F-8) 2. [On Creativity, Objectives, and Open-Endedness - Kenneth Stanley keynote at HLAI - YouTube](https://www.youtube.com/watch?v=y2I4E_UINRo) 3. Why Greatness Cannot be planned 8. Are LLMs creative papers? 1. - [https://towardsdatascience.com/exploring-creativity-in-large-language-models-from-gpt-2-to-gpt-4-1c2d1779be57](https://towardsdatascience.com/exploring-creativity-in-large-language-models-from-gpt-2-to-gpt-4-1c2d1779be57) 2. [https://arxiv.org/pdf/2304.00008.pdf](https://arxiv.org/pdf/2304.00008.pdf) 3. [https://www.perplexity.ai/search/c962a014-d831-4941-ab3e-5d3531ebd51b?s=u](https://www.perplexity.ai/search/c962a014-d831-4941-ab3e-5d3531ebd51b?s=u) 4. [https://garymarcus.substack.com/p/stop-treating-ai-models-like-people](https://garymarcus.substack.com/p/stop-treating-ai-models-like-people) 9. Arguing that AI is creative 1. [Prof. Chris Summerfield on creativity in large language models #machinelearning - YouTube](https://www.youtube.com/watch?v=4nJlqSEJTLA 3. [The AI Revolution: How Auto-GPT Unleashes a New Era of Automation and Creativity | by Sriram Parthasarathy | Towards AI](https://pub.towardsai.net/the-ai-revolution-how-auto-gpt-unleashes-a-new-era-of-automation-and-creativity-2008aa2ca6ae) 4. [Comments - What AI can do with a toolbox... Getting started with Code Interpreter](https://www.oneusefulthing.org/p/what-ai-can-do-with-a-toolbox-getting/comments) 5. sam altman tweet 10. Arguing that AI is not creative 1. [Imitation versus Innovation: What children can do that large language and language-and-vision models cannot (yet)](https://arxiv.org/pdf/2305.07666.pdf) 2. [Why AI is Harder Than We Think](https://arxiv.org/pdf/2104.12871.pdf) 11. Creativity 1. [The art and science of creativity](https://ia601808.us.archive.org/4/items/artofscientifici00beve/artofscientifici00beve_bw.pdf) 2. [Creativity and Consciousness - BRETT HALL](https://www.bretthall.org/creativity-and-consciousness.html) 3. [Critical and Creative Thinking - BRETT HALL](https://www.bretthall.org/critical-and-creative-thinking.html) TODO: Consider breaking out world models and analogy? ### Queue (Technical Background) 1. Transformer Intuitions 1. Andrej Karpathy Intuitions 1. [Tweet / Twitter](https://twitter.com/karpathy/status/1582807367988654081?lang=en) 2. [Let's build GPT: from scratch, in code, spelled out. - YouTube](https://www.youtube.com/watch?v=kCc8FmEb1nY) (also see my notes in intuitiveml) 2. [Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5](https://daleonai.com/transformers-explained) 3. [Transformer: A Novel Neural Network Architecture for Language Understanding – Google Research Blog](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) 4. [What Is ChatGPT Doing … and Why Does It Work?—Stephen Wolfram Writings](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/) 2. Transformer Implementations 1. [What is a Transformer? (Transformer Walkthrough Part 1/2) - YouTube](https://youtu.be/bOYE6E8JrtU?list=PL7m7hLIqA0hoIUPhC26ASCVs_VrqcDpAz&t=848) 2. [Implementing GPT-2 From Scratch (Transformer Walkthrough Part 2/2) - YouTube](https://youtu.be/dsjUDacBw8o?list=PL7m7hLIqA0hoIUPhC26ASCVs_VrqcDpAz) 3. [A Walkthrough of A Mathematical Framework for Transformer Circuits - YouTube](https://youtu.be/KV5gbOmHbjU?list=PL7m7hLIqA0hoIUPhC26ASCVs_VrqcDpAz) 4. [The Annotated Transformer](http://nlp.seas.harvard.edu/annotated-transformer/) 5. [Let's build GPT: from scratch, in code, spelled out. - YouTube](https://youtu.be/kCc8FmEb1nY) 3. [Survey of LLMs - https://arxiv.org/pdf/2303.18223v4.pdf](https://arxiv.org/pdf/2303.18223v4.pdf) 4. Mech Interp 1. [Mechanistic Interpretability Quickstart Guide — Neel Nanda](https://www.neelnanda.io/mechanistic-interpretability/quickstart) 2. [Concrete Steps to Get Started in Transformer Mechanistic Interpretability — Neel Nanda](https://www.neelnanda.io/mechanistic-interpretability/getting-started) 3. [An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers — Neel Nanda](https://www.neelnanda.io/mechanistic-interpretability/favourite-papers) 4. [Analogies between Software Reverse Engineering and Mechanistic Interpretability — Neel Nanda](https://www.neelnanda.io/mechanistic-interpretability/reverse-engineering) 5. [Toy Models of Superposition](https://transformer-circuits.pub/2022/toy_model/index.html) 6. [A Comprehensive Mechanistic Interpretability Explainer & Glossary - Dynalist](https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J) 7. [Actually, Othello-GPT Has A Linear Emergent World Representation — LessWrong](https://www.lesswrong.com/s/nhGNHyJHbrofpPbRG/p/nmxzr2zsjNtjaHh7x) 8. [Actually, Othello-GPT Has A Linear Emergent World Representation — Neel Nanda](https://www.neelnanda.io/mechanistic-interpretability/othello#results) 9. [200 Concrete Open Problems in Mechanistic Interpretability: Introduction — AI Alignment Forum](https://www.alignmentforum.org/posts/LbrPTJ4fmABEdEnLf/200-concrete-open-problems-in-mechanistic-interpretability) 10. [Neel Nanda Live MechInterp Research](https://twitter.com/neelnanda5/status/1682827872191348736?s=46) 8. General Transformer papers 1. [Challenges and Applications of Large Language Models](https://arxiv.org/pdf/2307.10169.pdf) 2. [https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) 3. [https://arxiv.org/pdf/2305.07185.pdf](https://arxiv.org/pdf/2305.07185.pdf) 4. [Tree of thoughts](https://arxiv.org/pdf/2305.10601.pdf) 5. [Memory Transformer](https://arxiv.org/pdf/2006.11527.pdf) 6. [DL for tabular data](https://m-clark.github.io/posts/2021-07-15-dl-for-tabular/) 7. [Latent ODEs](https://proceedings.neurips.cc/paper/2019/file/42a6845a557bef704ad8ac9cb4461d43-Paper.pdf) 8. [SAINT: Tabular row wise contrastive pretraining](https://arxiv.org/pdf/2106.01342.pdf) 9. [Transformers in Time Series: survey](https://arxiv.org/pdf/2202.07125.pdf) 10. [Temporal Fusion Transformers](https://arxiv.org/pdf/1912.09363.pdf) 11. General Deep Learning 1. [DeepMind x UCL | Deep Learning Lecture Series 2020 - YouTube](https://www.youtube.com/playlist?list=PLqYmG7hTraZCDxZ44o4p3N5Anz3lLRVZF) ### Queue (Indirectly Related) 2. Elon Musk Wait But Why Blog Post 3. Find book on 4. How did Einstein come up with General Relatively (look at his bio, poppers interpretation). 5. Thomas Kuhn paradigm shifts 6. To explain the world. 7. [https://twitter.com/daviddeutschoxf/status/1677649041105059842?s=46](https://twitter.com/daviddeutschoxf/status/1677649041105059842?s=46) 4. Theory of Constraints 5. ### To Generate More Ideas 1. Twitter/Perplexity Deep dive: ask if LLMs are creative (be able to Steel man the other side) 2. ### Other Resources 1. [AI Canon | Andreessen Horowitz](https://a16z.com/2023/05/25/ai-canon/)