Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1441 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0886. Min Act: -0.0000

Data Index: 772928 (C4 (Web Text))

Max Activating Token Index: 92

Text #1

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.1198. Min Act: -0.0000

Data Index: 1304571 (C4 (Web Text))

Max Activating Token Index: 275

Text #2

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0233. Min Act: -0.0000

Data Index: 1261672 (C4 (Web Text))

Max Activating Token Index: 92

Text #3

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0898. Min Act: -0.0000

Data Index: 360950 (C4 (Web Text))

Max Activating Token Index: 120

Text #4

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0486. Min Act: -0.0000

Data Index: 45863 (C4 (Web Text))

Max Activating Token Index: 124

Text #5

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9643. Min Act: -0.0000

Data Index: 377030 (C4 (Web Text))

Max Activating Token Index: 113

Text #6

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0469. Min Act: -0.0000

Data Index: 81656 (C4 (Web Text))

Max Activating Token Index: 155

Text #7

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9626. Min Act: -0.0000

Data Index: 1064828 (C4 (Web Text))

Max Activating Token Index: 76

Text #8

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0110. Min Act: -0.0000

Data Index: 255081 (C4 (Web Text))

Max Activating Token Index: 186

Text #9

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9238. Min Act: -0.0000

Data Index: 519000 (C4 (Web Text))

Max Activating Token Index: 170

Text #10

Max Range: 1.1198. Min Range: -1.1198

Max Act: 1.0048. Min Act: -0.0000

Data Index: 1329893 (C4 (Web Text))

Max Activating Token Index: 41

Text #11

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.8504. Min Act: -0.0000

Data Index: 642176 (C4 (Web Text))

Max Activating Token Index: 303

Text #12

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9050. Min Act: -0.0000

Data Index: 242571 (C4 (Web Text))

Max Activating Token Index: 744

Text #13

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9670. Min Act: -0.0000

Data Index: 810796 (C4 (Web Text))

Max Activating Token Index: 222

Text #14

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.8387. Min Act: -0.0000

Data Index: 543227 (C4 (Web Text))

Max Activating Token Index: 721

Text #15

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9625. Min Act: -0.0000

Data Index: 1024119 (C4 (Web Text))

Max Activating Token Index: 407

Text #16

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.8781. Min Act: -0.0000

Data Index: 1303829 (C4 (Web Text))

Max Activating Token Index: 875

Text #17

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9576. Min Act: -0.0000

Data Index: 1321639 (C4 (Web Text))

Max Activating Token Index: 206

Text #18

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9034. Min Act: -0.0000

Data Index: 1160129 (C4 (Web Text))

Max Activating Token Index: 282

Text #19

Max Range: 1.1198. Min Range: -1.1198

Max Act: 0.9155. Min Act: -0.0000

Data Index: 474684 (C4 (Web Text))

Max Activating Token Index: 468