VGO: A Multimodal Spiking Neural Network
Cognitive Architecture on a Single Consumer GPU

Abstract. We present VGO, a complete multimodal cognitive system implemented entirely in Spiking Neural Networks (SNN) running on a single consumer-grade GPU (NVIDIA RTX 3090, 24 GB). VGO integrates 39 SNN modules spanning auditory perception (CER = 8.1%), visual recognition (80-class COCO object detection, 7-class emotion), speech synthesis, episodic memory (50,000 entries), a 7-stage cognitive reasoning pipeline, curiosity-driven active learning, and embodied motor control via MuJoCo simulation. The entire system occupies 535 MB of VRAM (2.2% of available memory) and achieves end-to-end inference energy of 100–500 picojoules per sample — approximately 10¹⁰ times more efficient than frontier LLMs. VGO was developed by a single engineer in 30 days. We argue that SNN-native multimodal cognition represents a viable, radically more efficient alternative to the Large Language Model paradigm.

Development Timeline: 30 Days, 1 Person, 1 GPU

From zero to a 39-module multimodal cognitive system — every milestone on a single RTX 3090.

1. Introduction

The dominant paradigm in artificial intelligence — the autoregressive Transformer trained on trillions of tokens — has achieved remarkable capabilities at extraordinary cost. Training a frontier LLM is estimated to require over $100 million in compute; inference consumes approximately 10,000 joules per request. This energy profile is ten billion times higher than the biological neural circuits that evolution has refined over 500 million years.

We present VGO (Vectorized Growth Optimization), a brain-inspired cognitive architecture that demonstrates an alternative path. Built entirely on Spiking Neural Networks (SNN) with Leaky Integrate-and-Fire (LIF) neurons and Spike-Timing-Dependent Plasticity (STDP), VGO is the first system to achieve multimodal cognition — hearing, seeing, speaking, learning, and reasoning — within a single SNN framework running on consumer hardware. Perhaps most remarkably, VGO was built by a non-engineer with no computer science degree, working entirely through AI-assisted pair programming — proving that the barrier to frontier research is no longer credentials, but vision.

VGO is not a toy demonstration. It is a functioning cognitive system with:

• 39 active SNN modules covering 6 sensory modalities
• SNN-native speech recognition at CER = 8.1% (approaching Whisper's ~5%, with 127× fewer parameters)
• 7-stage cognitive pipeline with reflection, verification, and self-correction
• Curiosity-driven active learning with autonomous naming, spatial memory, and episodic recall
• Embodied motor control via SNN cerebellum connected to a MuJoCo humanoid
• 535 MB VRAM footprint on a single RTX 3090

2. Motivation: Why SNN, Why Now

2.1 The Energy Problem

The scaling laws of Transformer-based models project ever-increasing compute requirements. Epoch AI estimates that by 2030, training frontier models will require the energy output of a small power plant. This trajectory is physically unsustainable.

The human brain processes multimodal information on approximately 20 watts. A fruit fly navigates complex 3D environments with a brain consuming 10 microwatts. Both achieve this through spike-based computation: information is encoded in the timing of discrete electrical pulses, not in continuous-valued matrix multiplications.

2.2 The Learning Problem

LLMs cannot learn in real-time. When you tell ChatGPT your name, it forgets it next session. VGO achieves online learning through STDP — every interaction strengthens or weakens synaptic connections. Tell VGO "this is Wangcai" while showing it a dog, and it will recognize that dog by name the next time — without retraining, without fine-tuning.

2.3 Recent Context: The Eon Digital Fruit Fly

In March 2026, Eon Systems demonstrated a virtual fruit fly driven by 125,000 emulated neurons. While visually impressive, the Eon fly is a static simulation: it cannot learn, cannot adapt, and cannot generate novel behavior. VGO takes the opposite approach: rather than copying biology's structure, we adopt biology's learning rules.

3. System Architecture

┌─────────────────────────────────────────────────────┐
│           Client Layer (Flutter App / WebSocket)     │
├─────────────────────────────────────────────────────┤
│         Omni Server v3.0 (FastAPI, Mixin design)     │
│   /ask (text+image+audio)  /teach  /health  /ws      │
├─────────────────────────────────────────────────────┤
│                  Cognitive Core                       │
│  PARSE → ROUTE → RECALL → THINK → VERIFY →           │
│  SEARCH → RE-THINK → OUTPUT                          │
├─────────────────────────────────────────────────────┤
│              Sensory Perception Layer                 │
│  👂 CochleaEncoder → AuditoryCortex (384d)          │
│  👁️ RetinaEncoder → VisualCortex (4096d)           │
│  🗣 SNN CTC v2 + 3-gram LM (CER = 8.1%)            │
│  😊 EmotionSNN (7-class, audio×0.6 + visual×0.4)    │
│  📦 ObjectDetectorSNN (80-class COCO)                │
│  👤 FaceDetector + VoiceprintSNN (cross-modal)       │
├─────────────────────────────────────────────────────┤
│               Memory & Learning Layer                │
│  STDPCortex (4,096 neurons × 64 columns, 3.7M syn)  │
│  Hippocampus (50K entries, MiniLM 384d, QA-split)    │
│  EpisodicMemory (vision + audio + text binding)      │
│  CuriosityLoop (surprise > 0.5 → ask → learn)       │
├─────────────────────────────────────────────────────┤
│          Single NVIDIA RTX 3090 (535 MB / 24 GB)     │
└─────────────────────────────────────────────────────┘

4. Sensory Perception Layer

4.1 SNN Auditory System (CER = 8.1%)

VGO's hearing pipeline is fully SNN-native. The architecture: CochleaEncoder (64-channel Gammatone filterbank) → AuditoryCortex v15.4 (Conv1D spatiotemporal distillation, cos=0.545) → SNN CTC Decoder v2 (Conv1D+BiGRU+CTC, 6.0M params, AISHELL-1 trained) → 3-gram LM (beam search, CER 13%→8.1%, +37.2% relative improvement).

For comparison: Whisper Large-v3 achieves CER ~5% with 764M parameters (127× larger).

4.2 SNN Visual System

RetinaEncoder (224×224 DVS + Gabor V1) → VisualCortex (V1→V2→IT, 4096d, CLIP-distilled cos=0.75) → ObjectDetectorSNN (80 COCO classes, YOLO-distilled) → FaceDetectorSNN (512d face encoding) → ColorVision (128d opponent channels) → EmotionSNN (7-class, multimodal fusion).

4.3 SNN Speech Synthesis

Text → Pinyin → Mel (SNN Acoustic, 7.3M params) → Waveform (Vocos 24kHz, STFT=0.8529, 3× faster than HiFi-GAN).

5. Cognitive Reasoning Pipeline

Every text input passes through a 7-stage cognitive pipeline:

1. PARSE — Intent + entity + language detection
2. ROUTE — Simple→fast path; complex→slow path; math→SymbolicReasoner
3. RECALL — Semantic retrieval from 50K hippocampus with language-preference filter
4. THINK — Reflection-verification-correction loop (up to 3 iterations)
5. VERIFY — Hallucination detection via entropy + surprise thresholds
6. SEARCH — WebReflex v2.1: autonomous internet search when uncertain
7. RE-THINK → OUTPUT — KnowledgeDigester integrates search results, re-runs cognition

6. Memory and Learning Systems

6.1 STDPCortex

GPU-accelerated cortex: 4,096 LIF neurons × 64 columns, 3,751,300 synapses. Learning occurs through STDP modulated by surprise signals from PredictiveCoder.

6.2 Curiosity-Driven Active Learning Loop

VGO's most distinctive capability — the curiosity loop: Active observation (surprise > 0.5) → autonomous question → user answers → naming binding (regex extraction → tri-modal bind) → spatial memory (auto-deduplicate) → next encounter → episodic recall. This closed loop has no equivalent in any LLM or connectome-based system.

7. Embodied Intelligence: VGOBot

Dual-Brain Architecture: LLM Coach (cerebral cortex) provides high-level commands; SNN Cerebellum (61.1M params) converts to joint-level control. Training follows an 8-phase curriculum: static standing (98% success) → dynamic balance (87%) → walking → turning → squatting → running → grasping → stairs.

8. Performance Benchmarks

Energy Comparison

9. Comparison with Existing Systems

10. Limitations and Future Work

• Language generation: Retrieval-based only. SNN Decoder (Broca's Area) training in progress (val_loss=2.26).
• Reasoning depth: 3-step reflection vs. hundreds of CoT steps in LLMs.
• Knowledge scale: 50K entries vs. trillions of tokens. Addressable via continuous ingestion.
• SNN Acoustic v5 failure: PostNet batch-norm bug. v6 planned with fixed architecture.
• Embodied control: 98% static standing, but SNN cerebellum not yet independent without PPO support.

11. Conclusion

VGO demonstrates that multimodal cognition does not require the brute-force matrix multiplication paradigm that defines modern AI. A single engineer, working for 30 days on a single consumer GPU, has produced a cognitive system that:

• Hears Mandarin speech at 8.1% CER with a 6M-parameter SNN (vs. Whisper's 764M)
• Sees, recognizes, and names objects through autonomous curiosity
• Learns from single interactions through STDP — no retraining needed
• Reasons through a 7-stage cognitive pipeline with self-verification
• Controls a humanoid robot body through an SNN cerebellum
• Consumes 100–500 picojoules per inference — ten billion times less than mainstream LLMs

The implication is profound: intelligence is not a function of compute. It is a function of architecture. The biological brain proved this 500 million years ago. VGO is our attempt to relearn that lesson.

"They copied biology's wiring. We adopted biology's learning rules. The difference is between a photograph and a living organism."

VGO is open-source under the OPEN-Yongnian project. All code, weights, and training scripts are available at the project repository.

References

1. Maass, W. (1997). Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9).
2. Bi, G. & Poo, M. (1998). Synaptic modifications in cultured hippocampal neurons. J. Neuroscience, 18(24).
3. Shiu, P. et al. (2024). Whole-brain computational model of Drosophila melanogaster. Nature.
4. Davies, M. et al. (2018). Loihi: A neuromorphic manycore processor. IEEE Micro, 38(1).
5. Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.
6. Du, J. et al. (2024). SpikeVoice: Neural vocoder with spiking neural networks. ACL 2024.
7. Eon Systems (2026). The First Multi-Behavior Brain Upload. Technical Demonstration.