Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5808 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.3209. Min Range: -4.3209

Max Act: 4.3209. Min Act: -0.0000

Data Index: 1650262 (Python Code)

Max Activating Token Index: 329

Text #1

Max Range: 4.3209. Min Range: -4.3209

Max Act: 3.6024. Min Act: -0.0000

Data Index: 1531137 (Python Code)

Max Activating Token Index: 832

Text #2

Max Range: 4.3209. Min Range: -4.3209

Max Act: 3.5344. Min Act: -0.0000

Data Index: 1587012 (Python Code)

Max Activating Token Index: 897

Text #3

Max Range: 4.3209. Min Range: -4.3209

Max Act: 3.2547. Min Act: -0.0000

Data Index: 1398174 (Python Code)

Max Activating Token Index: 998

Text #4

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.9436. Min Act: -0.0000

Data Index: 999329 (C4 (Web Text))

Max Activating Token Index: 577

Text #5

Max Range: 4.3209. Min Range: -4.3209

Max Act: 3.0584. Min Act: -0.0000

Data Index: 1541432 (Python Code)

Max Activating Token Index: 536

Text #6

Max Range: 4.3209. Min Range: -4.3209

Max Act: 3.0127. Min Act: -0.0000

Data Index: 1397854 (Python Code)

Max Activating Token Index: 839

Text #7

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.7544. Min Act: -0.0000

Data Index: 1422737 (Python Code)

Max Activating Token Index: 408

Text #8

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.7269. Min Act: -0.0000

Data Index: 388148 (C4 (Web Text))

Max Activating Token Index: 146

Text #9

Max Range: 4.3209. Min Range: -4.3209

Max Act: 3.1232. Min Act: -0.0000

Data Index: 1561090 (Python Code)

Max Activating Token Index: 491

Text #10

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.5501. Min Act: -0.0000

Data Index: 1354728 (C4 (Web Text))

Max Activating Token Index: 8

Text #11

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.7830. Min Act: -0.0000

Data Index: 760374 (C4 (Web Text))

Max Activating Token Index: 639

Text #12

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.5022. Min Act: -0.0000

Data Index: 1368782 (Python Code)

Max Activating Token Index: 672

Text #13

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.7690. Min Act: -0.0000

Data Index: 1637411 (Python Code)

Max Activating Token Index: 389

Text #14

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.6347. Min Act: -0.0000

Data Index: 354141 (C4 (Web Text))

Max Activating Token Index: 249

Text #15

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.6153. Min Act: -0.0000

Data Index: 1409145 (Python Code)

Max Activating Token Index: 928

Text #16

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.5875. Min Act: -0.0000

Data Index: 1164705 (C4 (Web Text))

Max Activating Token Index: 875

Text #17

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.6721. Min Act: -0.0000

Data Index: 1365482 (Python Code)

Max Activating Token Index: 366

Text #18

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.5476. Min Act: -0.0000

Data Index: 1599647 (Python Code)

Max Activating Token Index: 288

Text #19

Max Range: 4.3209. Min Range: -4.3209

Max Act: 2.6310. Min Act: -0.0000

Data Index: 923517 (C4 (Web Text))

Max Activating Token Index: 461