Mamba - Linear Time Sequence Modeling with Selective State Spaces

I. Introduction

---

II. Background

---

III. Framework

---

III-I. Unsupervised Pre-training

---

III-II. Supervised Fine-tuning

---

III-III. Task-specific input Transformation

---

IV. Experiment

---

IV-I. Setup

---

Datasets

---

Model Specification

---

IV-II. Supervised Fine-tuning

---

Hyper-parameters

---

• LR : $6.25\times 10^{-5}$
  • lr-decay : .2 with warmup
  • $\lambda :$ .5
• batch-size : 32
• dropout : .1
• epochs : 3

Loss(Objective)

---

V. Analysis

---

V-I. Impact of number of layers transferred

---

V-II. Zero-shot Behavior

---

V-III. Ablation Study

---

VI. Conclusion

---

Contribution

---