Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1907 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.7177. Min Act: -0.0001

Data Index: 309986 (C4 (Web Text))

Max Activating Token Index: 637

Text #1

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.7037. Min Act: -0.0001

Data Index: 295856 (C4 (Web Text))

Max Activating Token Index: 417

Text #2

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.7740. Min Act: -0.0001

Data Index: 10384 (C4 (Web Text))

Max Activating Token Index: 98

Text #3

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6821. Min Act: -0.0001

Data Index: 1239648 (C4 (Web Text))

Max Activating Token Index: 47

Text #4

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.7526. Min Act: -0.0001

Data Index: 1063191 (C4 (Web Text))

Max Activating Token Index: 943

Text #5

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6979. Min Act: -0.0001

Data Index: 257621 (C4 (Web Text))

Max Activating Token Index: 110

Text #6

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6624. Min Act: -0.0001

Data Index: 938644 (C4 (Web Text))

Max Activating Token Index: 41

Text #7

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6941. Min Act: -0.0001

Data Index: 1234490 (C4 (Web Text))

Max Activating Token Index: 34

Text #8

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.5673. Min Act: -0.0001

Data Index: 1307329 (C4 (Web Text))

Max Activating Token Index: 540

Text #9

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6393. Min Act: -0.0001

Data Index: 50465 (C4 (Web Text))

Max Activating Token Index: 107

Text #10

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6645. Min Act: -0.0001

Data Index: 525040 (C4 (Web Text))

Max Activating Token Index: 644

Text #11

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6195. Min Act: -0.0001

Data Index: 575854 (C4 (Web Text))

Max Activating Token Index: 187

Text #12

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6040. Min Act: -0.0001

Data Index: 699696 (C4 (Web Text))

Max Activating Token Index: 151

Text #13

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6116. Min Act: -0.0001

Data Index: 247191 (C4 (Web Text))

Max Activating Token Index: 195

Text #14

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.5241. Min Act: -0.0001

Data Index: 749461 (C4 (Web Text))

Max Activating Token Index: 854

Text #15

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6335. Min Act: -0.0001

Data Index: 568682 (C4 (Web Text))

Max Activating Token Index: 1018

Text #16

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.5980. Min Act: -0.0001

Data Index: 1335386 (C4 (Web Text))

Max Activating Token Index: 549

Text #17

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6828. Min Act: -0.0001

Data Index: 860427 (C4 (Web Text))

Max Activating Token Index: 105

Text #18

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.5801. Min Act: -0.0001

Data Index: 78545 (C4 (Web Text))

Max Activating Token Index: 433

Text #19

Max Range: 0.7740. Min Range: -0.7740

Max Act: 0.6032. Min Act: -0.0001

Data Index: 1311514 (C4 (Web Text))

Max Activating Token Index: 184