Genomic Foundation Model
Developing a genomic foundation model that learns evolutionary patterns across multiple species using state-space model (Mamba SSM) architecture. The model captures long-range dependencies in genomic sequences more efficiently than traditional transformer approaches.
Problem
Existing genomic models are species-specific and fail to capture cross-species evolutionary patterns. Transformers are computationally expensive for long genomic sequences.
Solution
Building a multi-species foundation model using Mamba SSM architecture that scales linearly with sequence length and captures long-range dependencies across multiple genomes.
Architecture
- Mamba SSM backbone with selective state spaces
- Multi-species tokenizer (6-mer encoding)
- Pre-training on 50+ species genomes
- Fine-tuning heads for downstream tasks
- Distributed training with PyTorch DDP
- W&B experiment tracking
Results
In progress. Expected to outperform transformer-based models on downstream genomic tasks with 3x faster inference.
FAQs
Why Mamba SSM over Transformers?
Mamba SSM provides linear-time inference vs quadratic for Transformers, critical for long genomic sequences (up to 10M base pairs).