[AINews] Too Cheap To Meter: AI prices cut 50-70% in last 30 days • ButtondownTwitterTwitter

The AI Twitter Recap section provides a detailed overview of recent AI model developments, releases, performance benchmarks, tools, frameworks, research insights, and applications. It highlights key updates such as the release of new models like Llama3.1 405b and Sonnet 3.5, performance comparisons of Mistral Large and Claude-3.5, introduction of AI tools like FlexAttention in PyTorch API, discussions on Reinforcement Learning from Human Feedback (RLHF), compute-optimal scaling in large language models, model merging techniques, and the announcement of SAM 2 for Object Segmentation.

AI Reddit Recap

• Theme 1. Free Access to Advanced LLMs: Llama 3.1 405B and Sonnet 3.5

  • Google Cloud offers free access to Llama 3.1 405B and Sonnet 3.5 models, providing $300 worth of API usage.
  • Introduction of Llama3-s, a multimodal model trained on 1.4 trillion text tokens and 700 billion audio tokens.

• Theme 2. Optimized Inference and Quantization for ARM-based Processors

  • Snapdragon X CPU demonstrates fast inference with quantization for Llama 3.1 8B.
  • LG AI releases Exaone-3.0, a 7.8 billion parameter language model with superior performance.

• Theme 3. Summarization Techniques and Model Comparison for Large Texts

  • Discussion on best summarizing LLMs for consumer-grade hardware like Nvidia RTX 3060.
  • Mention of Gemini 1.5 Flash offering impressive summarization capabilities.

• Theme 4. Repurposing Mining Hardware for AI Workloads

  • User acquires a mining rig with 7x 3060 GPUs and seeks advice on loading AI models.
  • Recommendations include using LLaMA 3.1 70B on the rig's VRAM and upgrading motherboard and CPU for better performance.

• AI Model Improvements and Techniques

  • Flux with LoRA enhances photorealism; Midjourney to Runway video generation impresses.

• OpenAI Developments and Speculation

  • Teaser about 'Project Strawberry' and OpenAI working on new reasoning technology.

• AI Model Behavior and Limitations

  • ChatGPT struggles with letter counting, pointing out tokenization impact on model performance.

• Community Reactions and Discussions

  • Skepticism towards OpenAI's marketing and debate on AI progress.

BPF Insights for CUDA Profiling:

• Members discussed the use of BPF for CUDA profiling, noting limitations to the OS kernel.
• Suggestions were made for alternatives like Nsight Compute and Nsight Systems.

Attention Gym Links & FlexAttention:

• Malfunctioning link for Attention Gym was reported and discussion on integrating FlexAttention into HF models.

torchao v0.4.0 is Here:

• Enhancements like KV cache quantization and quantization aware training announced.
• Community engagement around low bit quantization experimentation.

Memory Usage and KV Cache Optimization:

• Member optimized KV Cache for memory usage and discussed code cleanup.

RoPE Optimization Discussions:

• Discussion on simplifying RoPE implementation for enhanced code clarity.

Discussions in this section cover various topics related to custom deployment, tool activation, and user experiences. Members shared code snippets for producing grounded answers and addressed concerns with Azure AI Search integration. Additionally, there were discussions about enabling tools by default in Cohere-toolkit, encountering hurdles in custom deployment, and the need for clarity in model feedback. The LlamaIndex Discord section announced upcoming events and highlighted concerns about observability in RAG pipelines. The section also covered discussions on LongRAG paper comparisons, self-routing techniques, and the effectiveness of Workflows in AI applications. Further topics included challenges with LLAMA 3 generation quality, memory optimization parameters, and plans for publicizing work in Discord channels like Torchtune and Modular (Mojo 🔥).

Discussions revolve around different AI models and their functionalities, including issues related to loss functions, token labels, logsumexp reduction, model loading, dataset processing, Hugging Face integration, inference optimization, and Colab limitations. Members seek clarification on token labels' meaning, question the necessity of logsumexp reduction, and explore solutions for processing smaller datasets and optimizing inference on A100 GPUs. Concerns are raised about Colab's disk space limitations and upgrading to Colab Pro. Links mentioned include Google Colab, huggingface datasets, and GitHub repositories for AI model development.

In the 'HuggingFace ▷ #<a href='https://discord.com/channels/879548962464493619/898619964095860757/1270839917597036659?utm_source=ainews&utm_medium=email&utm_campaign=ainews-too-cheap-to-meter-ai-prices-cut-50-70-in' target='_blank'>today-im-learning</a>' section, members are delving into topics like Neural Network Optimization, AI in Healthcare, and Embedding Serialization and Deserialization. Notable updates include the improvement of the model's loss through layer-wise scaling and a video discussing AI in Healthcare by Professor Andrew Janowczyk. In the 'HuggingFace ▷ #<a href='https://discord.com/channels/879548962464493619/897390579145637909/1271023723188584450?utm_source=ainews&utm_medium=email&utm_campaign=ainews-too-cheap-to-meter-ai-prices-cut-50-70-in' target='_blank'>cool-finds</a>' section, the dominance of Transformers, the EU's AI Act, and implications of staggered compliance deadlines are explored. Additionally, a discussion on the future of AI experimentation and the EU AI regulations is ongoing in the 'HuggingFace ▷ #<a href='https://discord.com/channels/879548962464493619/1156269946427428974/1271056966084464640?utm_source=ainews&utm_medium=email&utm_campaign=ainews-too-cheap-to-meter-ai-prices-cut-50-70-in' target='_blank'>reading-group</a>' part. A wealth of information is shared in the 'HuggingFace ▷ #<a href='https://discord.com/channels/879548962464493619/922424143113232404/1271017097106948208?utm_source=ainews&utm_medium=email&utm_campaign=ainews-too-cheap-to-meter-ai-prices-cut-50-70-in' target='_blank'>computer-vision</a>' section, covering resources from Papers with Code to IAM On-Line Handwriting Database. Lastly, in the 'HuggingFace ▷ #<a href='https://discord.com/channels/879548962464493619/922424173916196955/1271034157610303590?utm_source=ainews&utm_medium=email&utm_campaign=ainews-too-cheap-to-meter-ai-prices-cut-50-70-in' target='_blank'>NLP</a>' segment, discussions revolve around AutoProcessor availability and the features of InternLM 2.5. Each section reflects a vibrant exchange of knowledge and insights among the members.

This section covers various discussions related to PyTorch topics such as issues with Attention Gym link malfunction, plans for FlexAttention integration, Torch serialization complexities, device type for autocast in torch.autocast, and Flash Attention compatibility. Members discussed Torch serialization challenges and provided a workaround using model.compile() to avoid state dict complications. There were inquiries about device types for torch.autocast and compatibility of Flash Attention and Paged Attention. The section also includes links to resources such as torchao v0.4.0 release, Intx Tensor Subclasses Quantization, and issues related to model execution complexities in PyTorch.

Perplexity API Experiencing Major Outages:

• Users reported being unable to access the Perplexity API, with some noting a major outage according to the status page.
• Concerns were raised about the scope of the outage, with several users sharing their own experiences of failure to access the API.

Geo-Based Access Discrepancies:

• Some members suggested that the API outages might be geo-based, with users from different regions experiencing varying levels of access.
• A workaround using a VPN to Europe was proposed by one user.

Claude Outage May Affect API Functionality:

• Speculations arose that issues with the API might be linked to a Claude outage since users rely on it for processing results.
• Interdependencies between services were indicated to influence access to the Perplexity API.

Incoherence in Non-English Language Processing:

• Concerns were raised about incoherence in responses generated in non-English languages, highlighting issues with accurate translations and repetitive results.
• Questions were raised about the model's effectiveness in handling diverse languages and prompts.

Challenges with Google Maps URLs:

• One user shared difficulties in generating accurate Google Maps URLs for trip itineraries, indicating ongoing challenges with real-time data integration and result accuracy.

Gary Marcus Predictions:

• Gary Marcus predicts AI bubble collapse imminently.

Audience Capture:

• Members share humorous takes on audience capture in the AI community.

Contrarian Perspectives on AI:

• Diverse viewpoints on AI spark engaging discussions among members.

Mixed views on Marcus's insights

• Nathan described Gary Marcus as a bozo driven by irrelevant takes and a history of peculiar viewpoints, suggesting a level of audience capture in his commentary.

  • He voiced concerns, stating, “I have a hard time with people who make their career on tech but chronically hate tech.”

• Another member acknowledged that while Gary Marcus provides some sensible critiques regarding LLMs, he also has a penchant for contrarianism.

  • They noted that his genuine points get overshadowed by a desire to be right, seeking opportunities to proclaim, “I told you so.”

This section provides updates from the OpenAccess AI Collective Discord channel. Members discussed various topics such as training models with large datasets, prompt formatting for inference, LoRA import errors, configuration clarifications, and details on Llama 3 training. The community shared links related to Llama 3 models and text generation web UI tools. The section highlights collaborative efforts and knowledge sharing among members to address challenges and enhance model performance.

This section delves into discussions revolving around modular development tools and practices. Key points include the usage of VS Code with WSL for Mojo development on Windows, benefits and limitations of WSL, FancyZones utility for window management, and a debate on Active Directory as a distributed database. The conversations provide insights, suggestions, and experiences related to using these tools and technologies, aiming to enhance developers' workflows and efficiency.

Anthropic has addressed high error rates affecting their services, specifically on 3.5 Sonnet and 3 Opus. They have implemented a mitigation strategy, and as of Aug 8, 17:29 PDT, success rates have returned to normal levels. Access for Claude.ai free users has also been restored. Anthropic is closely monitoring the situation and providing updates as issues are resolved.