We're excited to share a recent paper that has caught our attention: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States". It introduces a novel approach to language modeling: Test-Time Training (TTT) layers. It presents a new class of sequence modeling layers that combine the efficiency of RNNs with the expressiveness of self-attention.

Key Innovations

1. TTT Layers: The core idea is to make the hidden state of an RNN a machine learning model itself, with the update rule being a step of self-supervised learning.

2. Linear Complexity: TTT layers maintain linear complexity.

3. Two Instantiations: The paper introduces TTT-Linear and TTT-MLP, where the hidden state is a linear model and a two-layer MLP, respectively.

Performance Highlights

• Both TTT-Linear and TTT-MLP match or exceed the performance of Transformer and Mamba (a modern RNN) baselines.

• TTT layers show strength in long context scenarios, continuing to reduce perplexity with more tokens, unlike Mamba.

• TTT-Linear is faster than Transformer at 8k context and matches Mamba in wall-clock time.

Technical Details

• The hidden state in TTT layers is updated using gradient descent on a self-supervised loss.

• The paper introduces "mini-batch TTT" and a "dual form" implementation.

• TTT layers can be integrated into existing network architectures.

Scientific Insights

The research highlights the advantage of TTT layers in improving the adaptability and efficiency of recurrent neural networks (RNNs). By integrating more expressive hidden states that learn at test time, these models can better capture and utilize context, leading to superior performance on tasks that involve long sequences. This innovation addresses a critical challenge in sequence modeling—efficiently managing long-range dependencies without excessive computational overhead.

Future Directions

This work opens up new possibilities for efficient language modeling, especially for tasks requiring long context understanding. There are several promising directions for future research, including:

• Exploring more sophisticated parameterizations of self-supervised tasks

• Further systems optimizations for even better efficiency

• Scaling to longer contexts (millions of tokens) and larger models

• More ambitious instantiations of the inner loop model

Dive Deeper: Discover the future of sequence modeling by exploring this pioneering research. Read the full paper.

Happy reading!

In today's issue:

• Interesting papers from the week

• Noteworthy open-source

• From around the web

• Featured blog article

1. Interesting papers

Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models

This paper proposes an optimization-free and zero fine-tuning framework that applies complex and non-rigid edits to a single real image via a text prompt. Using widely-available generic pre-trained text-to-image diffusion models, the authors demonstrate the ability to modulate pose, scene, background, style, color, and even racial identity in an flexible manner through a single target text detailing the desired edit.

→ Go to paper

Large Language Models Struggle to Learn Long-Tail Knowledge

This paper investigates the relationship between the knowledge memorized by large language models and the information in their pre-training datasets. The authors find that there is a strong correlation between a model's accuracy in answering a fact-based question and how many documents associated with that question were seen during pre-training. They also find that retrieval-augmentation can reduce the dependence on relevant document count.

→ Go to paper

Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding

This paper presents MinD-Vis, a system which can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations.

→ Go to paper

2. Noteworthy Open-Source

• Pandas Autoprofiler: automatically visualize Pandas dataframes in Jupyter notebooks.

• Diffusion Bias Explorer: allows to explore how the text-to-image models like Stable Diffusion v1.4 and DALLE-2 represent different professions and adjectives.

• BertViz: an interactive tool for visualizing attention in Transformer language models such as BERT, GPT2, or T5. It can be run inside a Jupyter.

3. From Around The Web

• visualizing 1,000,000,000 points

• Small rant about LLMs by Linus: don't ship the API to the user. Text generation is not the product!

• An AI assistant that creates graphics for presentations

  

4. Featured Blog Post

This is a new section of the newsletter in which I feature interesting posts from applied machine learning professionals. The goal is to share knowledge that you can directly use in your work.

Using Large Language Models for Data Labeling

By Zachariah Zhang

TLDR — We can leverage the text generation power of large language models like GPT3 to generate labeled data to use for supervised learning. We can do so using prompting, in which we give the LM a description of the task, some examples, and a new example to generate a label. The data generated from this is, in general, going to be noisy and of lower quality than human labels. However, the speed of data collection as well as the ability to use humans in the loop makes this an effective way to collect a significant amount of labeled data for tough to label tasks.

Thanks for reading this week's issue of the newsletter! If you found this issue useful, please share it along with your friends and colleagues.

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

This paper presents BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). The paper finds that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning.

→ Go to paper

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

The paper discusses how the CLIP model can be improved by using modern Hopfield networks to tackle the problem of explaining away. The new model, CLOOB, is shown to outperform CLIP in zero-shot transfer learning across all considered architectures and datasets.

→ Go to paper

Self-conditioned Embedding Diffusion for Text Generation

This paper proposes a new method for continuous diffusion models that overcomes the limitations of previous models. The new model, self-conditioned embedding diffusion, is more efficient on accelerator hardware and produces comparable results to standard autoregressive language models.

→ Go to paper

Hope you found this issue useful and informative. If so, please share it with your network!

In this week's issue:

1. Summaries of Research papers

2. Noteworthy open-source & datasets

3. ICYMI interviews

4. From around the web

1. Summaries of Research papers

ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations

This paper introduces ImageNet-X, a set of sixteen human annotations of factors such as pose, background, or lighting the entire ImageNet-1k validation set as well as a random subset of 12k training images. The paper investigates 2,200 current recognition models and study the types of mistakes as a function of model's (1) architecture, e.g. transformer vs. convolutional, (2) learning paradigm, e.g. supervised vs. self-supervised, and (3) training procedures, e.g., data augmentation.

→ Go to paper

TextCraft: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Text

This paper introduces TextCraft, a method for generating high-fidelity and diverse 3D shapes without the need for (text, shape) pairs for training. TextCraft uses CLIP and a multi-resolution approach to improve the fidelity of the generated shape. To improve shape diversity, TextCraft uses a discrete latent space which is modeled using a bidirectional transformer. TextCraft also uses a novel variant of classifier-free guidance to further improve the accuracy-diversity trade-off.

→ Go to paper

Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

This paper proposes an alternative mining-based approach for zero-shot learning that is more flexible and interpretable than prompting. This method outperforms prompting on a wide range of tasks when using comparable templates.

→ Go to paper

Large Language Models Are Human-Level Prompt Engineers

This paper presents Automatic Prompt Engineer (APE), a method for automatically generating and selecting natural language instructions for large language models (LLMs). APE treats the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. Experiments on 24 NLP tasks show that APE-engineered prompts outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 19/24 tasks.

→ Go to paper

Rickrolling the Artist: Injecting Invisible Backdoors into Text-Guided Image Generation Models

This paper discusses backdoor attacks against text-guided generative models. These attacks exploit the fact that many text-guided image generation models rely on pre-trained text encoders from external sources. By slightly altering an encoder, an attacker can trigger the model to generate images with pre-defined attributes or images following a hidden, potentially malicious description. The paper demonstrates the high effectiveness of these attacks and highlights that the injection process of a single backdoor takes less than two minutes.

→ Go to paper

Logits are predictive of network type

It is possible to predict which deep network has generated a given logit vector with accuracy well above chance. A classifier is trained on the logit vectors of the trained set of a dataset to map the logit vector to the network index that has generated it. Results are better with randomly initialized networks, but also generalize to pretrained networks as well as fine-tuned ones. Classification accuracy is higher using unnormalized logits than normalized ones.

→ Go to paper

Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions

This paper investigates the robustness of multimodal classifiers to cross-modal dilutions. The authors develop a model that generates additional dilution text that leads to misclassification of the multimodal input. Experiments on two tasks show that the performance of task-specific fusion-based multimodal classifiers drops significantly in the presence of dilutions generated by the model.

→ Go to paper

Evaluating and Improving Factuality in Multimodal Abstractive Summarization

This paper proposes a new metric, CLIPBERTScore, for evaluating the factuality of abstractive document summarization, which takes into account the vision modality. This metric is a combination of CLIPScore and BERTScore, and is designed to leverage the robustness and strong factuality detection performance between image-summary and document-summary. The paper shows that this new metric outperforms existing factuality metrics for document summarization, and performs competitively with strong multimodal factuality metrics specifically fine-tuned for the task.

→ Go to paper

The Path to Autonomous Learners

This paper presents a new theoretical approach for enabling domain knowledge acquisition by intelligent systems. A hybrid model is introduced that starts with minimal input knowledge in the form of an upper ontology of concepts, stores and reasons over this knowledge through a knowledge graph database and learns new information through a Logic Neural Network. The behavior of this architecture when handling new data is studied and it is shown that the final system is capable of enriching its current knowledge as well as extending it to new domains.

→ Go to paper

2. Noteworthy open-source & datasets

• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

• Focal Modulation Networks: an attention-free architecture that achieves superior performance than SoTA self-attention

• Warp: Python framework for writing high-performance simulation and graphics code

• The stack: contains over 3TB of permissively-licensed source code files covering 30 programming languages crawled from GitHub

• ImageNet-X is a set of human annotations pinpointing failure types for the popular ImageNet dataset

  ---

3. ICYMI interviews

• OpenAI's Greg Brockman: The Future of LLMs, Foundation & Generative Models (DALL·E 2 & GPT-3)

• Codex demo: solving complex problems with multiple iterations (minute ~4:30)

4. From around the web

• PhotoGuard: Defending Against Diffusion-based Image Manipulation

  

• CLIP Interrogator 2.1: figure out what a good prompt might be to create new images like an existing one

  

• Run GPT-3 prompts in Google Sheets

Thank you for reading this week's newsletter! I hope you found it useful and that you learned something new. I'd love to know if you have feedback on any topic about which you'd like me to write more about or you just like to share some thoughts.