Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3234 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.7074. Min Act: -0.0000

Data Index: 930053 (C4 (Web Text))

Max Activating Token Index: 388

Text #1

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.8312. Min Act: -0.0000

Data Index: 439093 (C4 (Web Text))

Max Activating Token Index: 725

Text #2

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.8308. Min Act: -0.0000

Data Index: 988573 (C4 (Web Text))

Max Activating Token Index: 393

Text #3

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.3492. Min Act: -0.0000

Data Index: 122514 (C4 (Web Text))

Max Activating Token Index: 299

Text #4

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.3286. Min Act: -0.0000

Data Index: 133078 (C4 (Web Text))

Max Activating Token Index: 1008

Text #5

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.4868. Min Act: -0.0000

Data Index: 1237289 (C4 (Web Text))

Max Activating Token Index: 972

Text #6

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.4949. Min Act: -0.0000

Data Index: 750835 (C4 (Web Text))

Max Activating Token Index: 67

Text #7

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.2856. Min Act: -0.0000

Data Index: 309585 (C4 (Web Text))

Max Activating Token Index: 140

Text #8

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.4438. Min Act: -0.0000

Data Index: 648259 (C4 (Web Text))

Max Activating Token Index: 546

Text #9

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.3987. Min Act: -0.0000

Data Index: 541645 (C4 (Web Text))

Max Activating Token Index: 678

Text #10

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.3013. Min Act: -0.0000

Data Index: 39947 (C4 (Web Text))

Max Activating Token Index: 276

Text #11

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.2623. Min Act: -0.0000

Data Index: 375424 (C4 (Web Text))

Max Activating Token Index: 535

Text #12

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.4326. Min Act: -0.0000

Data Index: 1312199 (C4 (Web Text))

Max Activating Token Index: 537

Text #13

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.0291. Min Act: -0.0000

Data Index: 816986 (C4 (Web Text))

Max Activating Token Index: 150

Text #14

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.2077. Min Act: -0.0000

Data Index: 812398 (C4 (Web Text))

Max Activating Token Index: 941

Text #15

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.1534. Min Act: -0.0000

Data Index: 181250 (C4 (Web Text))

Max Activating Token Index: 851

Text #16

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.1296. Min Act: -0.0000

Data Index: 1085914 (C4 (Web Text))

Max Activating Token Index: 819

Text #17

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.3024. Min Act: -0.0000

Data Index: 118822 (C4 (Web Text))

Max Activating Token Index: 123

Text #18

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.0819. Min Act: -0.0000

Data Index: 711197 (C4 (Web Text))

Max Activating Token Index: 2

Text #19

Max Range: 6.8312. Min Range: -6.8312

Max Act: 6.0819. Min Act: -0.0000

Data Index: 606703 (C4 (Web Text))

Max Activating Token Index: 2