Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 957 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.2680. Min Range: -2.2680

Max Act: 2.2680. Min Act: -0.0000

Data Index: 419802 (C4 (Web Text))

Max Activating Token Index: 389

Text #1

Max Range: 2.2680. Min Range: -2.2680

Max Act: 2.1769. Min Act: -0.0000

Data Index: 1012540 (C4 (Web Text))

Max Activating Token Index: 388

Text #2

Max Range: 2.2680. Min Range: -2.2680

Max Act: 2.1370. Min Act: -0.0000

Data Index: 691169 (C4 (Web Text))

Max Activating Token Index: 91

Text #3

Max Range: 2.2680. Min Range: -2.2680

Max Act: 2.0783. Min Act: -0.0000

Data Index: 1236912 (C4 (Web Text))

Max Activating Token Index: 297

Text #4

Max Range: 2.2680. Min Range: -2.2680

Max Act: 2.0156. Min Act: -0.0000

Data Index: 892469 (C4 (Web Text))

Max Activating Token Index: 547

Text #5

Max Range: 2.2680. Min Range: -2.2680

Max Act: 2.0161. Min Act: -0.0000

Data Index: 1318944 (C4 (Web Text))

Max Activating Token Index: 343

Text #6

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7979. Min Act: -0.0000

Data Index: 280877 (C4 (Web Text))

Max Activating Token Index: 808

Text #7

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7068. Min Act: -0.0000

Data Index: 361949 (C4 (Web Text))

Max Activating Token Index: 917

Text #8

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7521. Min Act: -0.0000

Data Index: 947895 (C4 (Web Text))

Max Activating Token Index: 800

Text #9

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7207. Min Act: -0.0000

Data Index: 150404 (C4 (Web Text))

Max Activating Token Index: 475

Text #10

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7378. Min Act: -0.0000

Data Index: 517093 (C4 (Web Text))

Max Activating Token Index: 304

Text #11

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7044. Min Act: -0.0000

Data Index: 18314 (C4 (Web Text))

Max Activating Token Index: 471

Text #12

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.6964. Min Act: -0.0000

Data Index: 785053 (C4 (Web Text))

Max Activating Token Index: 1018

Text #13

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.5367. Min Act: -0.0000

Data Index: 109395 (C4 (Web Text))

Max Activating Token Index: 226

Text #14

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.6839. Min Act: -0.0000

Data Index: 1267248 (C4 (Web Text))

Max Activating Token Index: 360

Text #15

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7597. Min Act: -0.0000

Data Index: 1160143 (C4 (Web Text))

Max Activating Token Index: 281

Text #16

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.6244. Min Act: -0.0000

Data Index: 1209524 (C4 (Web Text))

Max Activating Token Index: 61

Text #17

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.7192. Min Act: -0.0000

Data Index: 640470 (C4 (Web Text))

Max Activating Token Index: 1021

Text #18

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.6198. Min Act: -0.0000

Data Index: 987573 (C4 (Web Text))

Max Activating Token Index: 385

Text #19

Max Range: 2.2680. Min Range: -2.2680

Max Act: 1.6204. Min Act: -0.0000

Data Index: 103155 (C4 (Web Text))

Max Activating Token Index: 1002