Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 889 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0493. Min Range: -1.0493

Max Act: 1.0493. Min Act: -0.0000

Data Index: 1054174 (C4 (Web Text))

Max Activating Token Index: 767

Text #1

Max Range: 1.0493. Min Range: -1.0493

Max Act: 1.0272. Min Act: -0.0000

Data Index: 1278333 (C4 (Web Text))

Max Activating Token Index: 654

Text #2

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9909. Min Act: -0.0000

Data Index: 1185489 (C4 (Web Text))

Max Activating Token Index: 547

Text #3

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9490. Min Act: -0.0000

Data Index: 113508 (C4 (Web Text))

Max Activating Token Index: 84

Text #4

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9633. Min Act: -0.0000

Data Index: 811168 (C4 (Web Text))

Max Activating Token Index: 416

Text #5

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9733. Min Act: -0.0000

Data Index: 1100686 (C4 (Web Text))

Max Activating Token Index: 272

Text #6

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9903. Min Act: -0.0000

Data Index: 177370 (C4 (Web Text))

Max Activating Token Index: 794

Text #7

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9345. Min Act: -0.0000

Data Index: 457470 (C4 (Web Text))

Max Activating Token Index: 1002

Text #8

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8598. Min Act: -0.0000

Data Index: 1161157 (C4 (Web Text))

Max Activating Token Index: 668

Text #9

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.9107. Min Act: -0.0000

Data Index: 108057 (C4 (Web Text))

Max Activating Token Index: 206

Text #10

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8268. Min Act: -0.0000

Data Index: 82361 (C4 (Web Text))

Max Activating Token Index: 533

Text #11

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8990. Min Act: -0.0000

Data Index: 1228345 (C4 (Web Text))

Max Activating Token Index: 822

Text #12

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8379. Min Act: -0.0000

Data Index: 1113853 (C4 (Web Text))

Max Activating Token Index: 845

Text #13

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8557. Min Act: -0.0000

Data Index: 50718 (C4 (Web Text))

Max Activating Token Index: 124

Text #14

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.7813. Min Act: -0.0000

Data Index: 102784 (C4 (Web Text))

Max Activating Token Index: 126

Text #15

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8083. Min Act: -0.0000

Data Index: 1071170 (C4 (Web Text))

Max Activating Token Index: 579

Text #16

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.8611. Min Act: -0.0000

Data Index: 986793 (C4 (Web Text))

Max Activating Token Index: 482

Text #17

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.7886. Min Act: -0.0000

Data Index: 728977 (C4 (Web Text))

Max Activating Token Index: 599

Text #18

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.7864. Min Act: -0.0000

Data Index: 689401 (C4 (Web Text))

Max Activating Token Index: 211

Text #19

Max Range: 1.0493. Min Range: -1.0493

Max Act: 0.7561. Min Act: -0.0000

Data Index: 1233047 (C4 (Web Text))

Max Activating Token Index: 406