[AINews] Nvidia Minitron: LLM Pruning and Distillation updated for Llama 3.1 • ButtondownTwitterTwitter

This section provides a recap of updates and developments related to AI discussed on Twitter and Reddit. The AI Twitter Recap highlights recent releases, updates, insights, and tools shared by various individuals and organizations in the AI community. It covers topics such as AI model launches, research techniques, applications, industry trends, and more. The AI Reddit Recap focuses on discussions from the LocalLlama subreddit, detailing updates on ExLlamaV2 with tensor parallel support and integration with TabbyAPI. The community's response to these developments, including performance improvements and bug fixes, is also highlighted.

AI Model Releases and Benchmarks

• AI21 Labs launched Jamba 1.5 Mini and Jamba 1.5 Large on the SSM-Transformer architecture. Jamba 1.5 Large achieved a score of 65.4 on Arena Hard outperforming other models.

• Grok 2 and its mini variant have been added to the LMSYS leaderboard, ranking #2.

• SmolLM models in sizes 135M, 360M, and 1.7B parameters have been released, trained on the Cosmo-Corpus dataset.

AI Development Tools and Frameworks

• Aider 0.52.0 introduced shell command execution and new features like ~ expansion and model switch to gpt-4o-2024-08-06.

• Cursor raised $60M for AI-powered coding, aiming to revolutionize software development.

• LangChain Python Documentation provided strategies to improve SQL query generation using create_sql_query_chain.

AI Research and Technical Advancements

• Mamba 2.8B model, a transformers-compatible language model, is released, requiring essential installations for optimized CUDA kernels.

• AutoToS automates planning with LLMs through a defined search space and feedback from unit tests.

• Multimodal LLM directly understands text and speech without a separate ASR stage, enabling faster responses.

The section discusses various advancements and aspirations in AI-powered code creation and planning processes using Language Model Machines (LLMs). "Cursor's Vision for AI-Powered Code Creation" and "LLMs in Planning: AutoToS Paper Insights" highlight the goals of developing AI-powered code editors for automating code tasks and optimizing planning processes with LLMs. Additionally, it covers updates on projects like Autogen, Cursor AI, California's AI regulations, and upcoming AI Engineer Meetup. Furthermore, it mentions discussions on taxonomy synthesis, model rankings, and open-source dilemmas in different Discord channels.

• Seek Open Interpreter Brand Guidelines: A user inquired about the availability of Open Interpreter brand guidelines, indicating a need for clarity on branding.
  • Could you share where to find those guidelines?
• Surprising Buzz Around Phi-3.5-mini: Users expressed unexpected approval for the performance of Phi-3.5-mini, sparking discussions that brought Qwen2 into the spotlight.
  • The positive feedback caught everyone off guard!
• Python Script Request for Screen Clicks: A user sought a Python script capable of executing clicks on specified screen locations based on text commands, like navigating in Notepad++.
  • How do I make it click on the file dropdown?
• --os mode Could Be a Solution: In response to the script query, it was suggested that using the --os mode might solve the screen-clicking challenge.
  • This could streamline operations significantly!
• Exciting Announcement for Free Data Analytics Masterclass: A user shared an announcement for a free masterclass on Data Analytics, promoting real-world applications and practical insights.
  • Interested participants can register here and share in the excitement over potential engagement.

Flux Pipeline Compilation Performance: When using torch.compile in FluxPipeline, performance can be slower than without it; the compilation happens in FluxPipeline's __init__ after input and weight scales are adjusted.### Fp8 Checkpoints for Flux Schnell: An fp8 checkpoint for Flux Schnell is available, and it's easy to create one by loading the pipeline and running at least 30 steps. This takes 6 minutes currently, and the code needs to be updated to handle loading from prequantized t5's.### Loading Time Improvements: Loading the pipeline takes 6 minutes, and the speed may be impacted by HF downloads. The author suggests allowing loading from prequantized t5's, which could be achieved by downloading a snapshot of the BFL HF weights.### Hugging Face Snapshot Downloads: A suggestion was made to allow users to download a snapshot of the BFL HF weights using huggingface_hub.download_snapshot(bfl/schnell).### Stable Diffusion Installation and Guides: A user asks for recommendations for guides on installing Stable Diffusion, and Automatic1111 and ComfyUI are suggested. It is noted that while AMD cards can be used for Stable Diffusion, they will be slower and the Tech Support channel provides helpful resources.

Aider 0.52.0 Released:

Aider 0.52.0 brings shell command execution, allowing users to launch a browser, install dependencies, run database migrations, exercise code changes, and run tests directly within the tool. Other key updates include ~ expansion for /read and /drop, a new /reset command to clear chat history, improvements to auto commit sequencing, and a default OpenAI model switch to gpt-4o-2024-08-06.

Aider Wrote 68% of the Code for This Release:

Aider autonomously generated 68% of the code for version 0.52.0, showcasing its growing capabilities in software development.

The 'Latent Space ▷ #ai-in-action-club' section discusses various topics including handling duplicate or similar topics, taxonomy synthesis, and using embedding models for topic similarity. It explores tools like Taxonomy Synthesis, GPT Researcher, and Embedland for AI applications. Additionally, a tool called Storm for hierarchical planning and BERTopic's algorithm for topic representation are mentioned. The discussion delves into methods for dealing with duplicate topics, hierarchical planning using taxonomy synthesis, and the efficiency of embedding models for topic similarity.

LORA + FSDP, suggesting that an 8xH100 configuration is necessary.

• They also mentioned issues with the tqdm progress bar being inaccurate when warm restarts are enabled during training.

Pretraining Doesn't Require Prompt Style: A member confirmed that pretraining does not necessitate a prompt style, implying that it can be done without specific input prompts.

• This was affirmed by another member, suggesting that the model's primary focus during pretraining is not on prompt engineering but on learning general patterns and representations from the data.

Structured Pretraining for Better Data Focus: A member pointed out that adding structure to pre-training data, such as including URLs at the start, can prevent overfitting on irrelevant information.

• They suggested that incorporating a system prompt with relevant information about the data could improve performance, but acknowledged that this technique has not been widely adopted and its effectiveness remains uncertain.

This section includes a discussion about REST API testing.

This section discusses various user queries related to preparing test pairs for REST API, clarification on 'executable test pairs,' and details about AI21 Labs' Jamba updates. It includes a user seeking guidance on creating test pairs, another user seeking clarification on test pair execution, and the announcement of Jamba 1.5 Mini & Large releases with superior features such as long context handling, speed, and quality. Additionally, Jamba's multilingual capabilities, developer-readiness, and accessibility are highlighted. The section also covers AI21 Labs events, including the NVIDIA AI Summit India, an AI Capabilities and Risks Demo-Jam Hackathon, and a Tinygrad compilation quest.