[AINews] Gemma 2 tops /r/LocalLlama vibe check • ButtondownTwitterTwitter

AI Twitter Recap

• Andrej Karpathy's new AI+Education company Eureka Labs

  • @karpathy announced he is starting an AI+Education company called Eureka Labs to build an AI native school. The goal is to make it easy for anyone to learn anything, with AI Teaching Assistants supporting human teachers. Their first product will be LLM101n, an undergraduate-level class on training your own AI. Course materials will be free, with revenue from digital/physical cohorts.
  • @DrJimFan noted that no one is more qualified to do EdTech than Andrej, and other AI startups in this area can't compete. He's glad they both like the name "Eureka".
  • @danielhanchen is excited for the LLM101n course, with chapters covering bigrams, attention, transformers, optimization, datasets, inference, fine-tuning, and deployment. He notes Andrej's course materials like CS231n and Zero to Hero are pure gold.

• New model releases

  • @GuillaumeLample announced the release of Mathstral 7B and Codestral Mamba 7B under Apache 2 license. Mathstral 7B obtains 56.6% pass@1 on MATH, outperforming Minerva 540B by 20%+. Codestral Mamba is one of the first open source models with a Mamba 2 architecture, the best 7B code model available.
  • @LoubnaBenAllal1 introduced SmolLM, a series of 135M, 360M and 1.7B models outperforming MobileLLM, Phi1.5 and Qwen2 small models. Trained on SmolLM-corpus of high quality web, code and synthetic data.
  • [@AnthropicAI](https://twitter.com/AnthropicAI/status/181323775

The AI Reddit Recap section covers various themes and discussions related to new model releases, model performance and limitations, AI hype versus long-term potential, image and video generation, AI model architectures, and AI regulation and public perception. Each theme includes posts from different subreddits highlighting advancements, challenges, humorous observations, and critical analysis within the AI community.

This section provides insights into recent developments and discussions within various AI-focused Discord channels. It covers topics such as advancements in AI model development, challenges in AI infrastructure, new benchmarks, and debates on AI-related issues. From Codestral Mamba's linear time inference model to LAION's cybersecurity concerns, the content delves into a wide array of AI-related subjects. The section highlights cutting-edge projects, open-source initiatives, and challenges faced by developers in the AI community.

The web page continues to highlight various technical discussions and insights shared across different Discord channels focused on AI, technology, and programming. The interactions delve into topics such as training distinctive illustration styles, troubleshooting technical issues with specific hardware needs, overcoming CUDA kernel call errors, debates on prompt variations and model capabilities, and exploration of innovative AI models and technologies. These discussions showcase a community engaged in sharing knowledge, exchanging ideas, and addressing challenges in the realm of artificial intelligence and tech development.

• Hannah Hype: Custom AI Assistant: An AI assistant, Hannah, is introduced with extreme customization features and integration with popular AI APIs like OpenAI and Anthropic.
• MongoDB Melds with LangChain for Hybrid Search: Discussions on using MongoDB for Hybrid Search and the demand for community insights on integrating with LangChain.
• AI's Answer to Viral Sports Videos: Community interest in AI tools for creating viral sports content like YouTube shorts/TikToks and exploring AI's capability in this area.
• Unstructured to Structured: LangChain's Document Conversion: Conversations on transforming unorganized data into LangChain documents for improved performance.
• Codestral's Code Conquest: Introduction of Codestral Mamba for advanced code productivity and reasoning.
• Mathstral: The Missing Model Mystery: Curiosity around a model named Mathstral and its potential association with Mistral AI.
• Curbing Overfitting: On the Hunt for Solutions: Suggestions for combatting overfitting with strategies like increasing rank and tweaking learning rates.
• Handheld Hardware Huzzah for My Friend V1: Excitement about the compact form factor of My Friend V1.
• Transcription Trust Talks for AI Friend: Privacy concerns regarding transcription interactions with AI Friend and the importance of confidentiality.
• Mac M3 Microchip Mystery with Open Interpreter: Questions on compatibility between Open Interpreter and M3 Mac.
• Torchtune v0.2.0 Unpacked: Release of Torchtune v0.2.0 with new models and features like sample packing.
• LLAMA 3's Finetuning Quirk: Issues with LLAMA 3's finetuning and suggestions to switch to stable releases.
• Linearizer Out, Updates In: Queries about updated notes post-removal of tinygrad's linearizer and the community's interest in documentation clarity.
• Color Code Conundrum Clarified: Clarification on message format nuances regarding color coding in a member's notes.

The section highlights the availability of Chilean touristic data for testing and analysis purposes. Additionally, it introduces various tools like Phi-3 Vision for Apple Silicon, Fast Subtitle Maker for Videos, and a YouTube Video Transcription Tool. These tools offer features like locally-run Vision and Language Models for Apple Silicon, quick generation of subtitles using Groq API's model, and transcription of YouTube videos using Deepgram and Claude, customizable with templates. The subsections cover discussions on learning by implementing papers, Inception model and ResNet, system requirements for AI stability models, prompt engineering for video generation, and the shared Skin Cancer Classification Project. There are also insights into extracting attention features from GhostNetV2, and discussions on RAG vs. fine-tuning models for specific tasks. The section additionally includes updates on the release of models like Mamba-Codestral-7B-v0.1 and Mathstral-7B-v0.1, Google's FlAMe 24B model, and insights into Llama 3's release and Kaggle Notebook handling issues.

The discussion in this section revolves around memory optimization in neural networks, particularly focusing on the comparison between different optimizers and their impact on training efficiency. Members debated the significance of 50% less memory usage compared to AdamW in neural networks and its implications for large-scale training. The heavy impact of optimizer costs on training was highlighted, with claims that AdamW is 3x more expensive than gradients, leading to potential doubling of batch sizes. The efficiency of the AdamW-Mini optimizer was confirmed to offer roughly 50% savings in memory usage. Concerns were raised regarding the distribution of costs and the impact on training large datasets. Strategies for handling multiple tables in Excel were also discussed.

This section delves into the performance and benchmarking aspects of Modular (Mojo 🔥) platform. It discusses the differences between 'parallelize' and 'sync_parallelize' functions, highlighting the importance of memory management for optimizing draft versions. Additionally, it addresses installation issues and the need for improvements in user experience. The section also covers the release of a new Mojo nightly compiler, improvements in SIMD optimization, and requests for more verbose reporting for MAX. Furthermore, insights into core utilization in Mojo, disparities in NumPy performance due to different BLAS backends, and the significance of manual timing in benchmarking are explored. The section concludes by emphasizing the superior performance of Intel MKL and the relevance of accurate dataset design for training language models.

Eleuther ▷ #research

• Standard deviation calculation debate: Discussion on calculating standard deviation in one pass versus the traditional two-pass approach.
  • One user mentioned implementing it but faced issues with the launch config.
• Debunking TransformerEngine claims: Users debated the fusion implementation of TransformerEngine, confirming it doesn't fuse normalization and linear layers as previously assumed. RMSNorm fusion was discussed as a superior method.
• Reformer: Efficient Transformer Clarifications: Highlights on Reformer and differences in attention matrices. Discussion on the lack of adoption of LSH attention.
• Challenges of Efficient Attention mechanisms: Reproducibility issues with Efficient Transformers like Reformer, mentioning potential success of Linear Transformers in addressing complexity.
• PLMs for distinguishing viral mimicry: User shared an accepted poster presentation using Protein Language Models to identify viral proteins mimicking human proteins with a 99.7% RO CAUC.

This section discusses various topics related to CUDA Mode Torch with 33 messages. It includes discussions on PyTorch profiler performance, Thunder vs Torch compile integration, nvfuser vs Triton performance comparison, concerns about custom kernel compilation time, and optimizing kernel compilation using nvfuser and Triton. Members exchange insights on improving performance, solving errors, and exploring different aspects of CUDA Mode Torch.

This section covers various discussions in the Interconnects channel, including topics like lobbying controversy, AI legislation polling, public perception of AI tools, GPT-4o vs Llama 405 tokenizers, Dell's marketing move, and critiques of AI research papers. Members debate over policy loss function overfitting concerns and the necessary degenerate case in DPO-like algorithms. The conversation also delves into preferences in sampling methods for policy models, challenges in DPO objectives, and insights from papers like Nemotron and Zephyr. Overall, the discussions range from AI industry controversies to humorous memes shared within the group.

The LlamaIndex section discusses various updates including the introduction of LlamaIndex and its agentic capabilities, improvements in LlamaParse for better handling of complex tables, the release of a new multi-agent tree system for managing customer interactions, customized AI solutions with LlamaIndex consulting services, and how Scaleport AI is using LlamaCloud and LlamaIndex to accelerate AI development. Each update includes details on the specific enhancements and benefits provided by LlamaIndex.

Members in this section discussed various topics related to AI models, detection services, voice extraction, and other AI-related issues. Some highlights include using moderation models to enhance focus and relevance in chatbots, concerns over exorbitant pricing for detection services, queries about voice extraction from podcasts, limitations of GPTs agents, issues with PUT actions in custom GPTs, problems with vector store embeddings not recognizing names, and errors due to exceeding API quotas for GPT-3.5 Turbo. Additionally, discussions covered AI tools for viral content creation, converting unstructured data into LangChain documents, contributing to the LangChain open-source community, handling errors in Qdrant, and implementing hybrid search in MongoDB with LangChain. Team collaborations, release announcements, and technical issues were also prominent in the forum discussions.

This section highlights various updates and discussions on topics related to AI and technology. It includes discussions on OpenAI access requirements, using LLM for hospital billing checks, generating Python code for bill checking, LLM finetuning, developer opportunities in HLS and WebRTC, Phoenix 2.0 product update and town hall event, new updates to AI21 Python SDK with Jamba-Instruct support, and async client support across all platforms.