Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5232 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.6289. Min Act: -0.0000

Data Index: 152949 (C4 (Web Text))

Max Activating Token Index: 176

Text #1

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.2963. Min Act: -0.0000

Data Index: 1106114 (C4 (Web Text))

Max Activating Token Index: 878

Text #2

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.2718. Min Act: -0.0000

Data Index: 1101277 (C4 (Web Text))

Max Activating Token Index: 950

Text #3

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.2314. Min Act: -0.0000

Data Index: 1257088 (C4 (Web Text))

Max Activating Token Index: 770

Text #4

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.2126. Min Act: -0.0000

Data Index: 756102 (C4 (Web Text))

Max Activating Token Index: 179

Text #5

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.1513. Min Act: -0.0000

Data Index: 272874 (C4 (Web Text))

Max Activating Token Index: 864

Text #6

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.9745. Min Act: -0.0000

Data Index: 590382 (C4 (Web Text))

Max Activating Token Index: 757

Text #7

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.9357. Min Act: -0.0000

Data Index: 1050280 (C4 (Web Text))

Max Activating Token Index: 367

Text #8

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.0274. Min Act: -0.0000

Data Index: 473725 (C4 (Web Text))

Max Activating Token Index: 625

Text #9

Max Range: 2.6289. Min Range: -2.6289

Max Act: 2.0869. Min Act: -0.0000

Data Index: 825792 (C4 (Web Text))

Max Activating Token Index: 678

Text #10

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.9307. Min Act: -0.0000

Data Index: 1587486 (Python Code)

Max Activating Token Index: 631

Text #11

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8248. Min Act: -0.0000

Data Index: 1102870 (C4 (Web Text))

Max Activating Token Index: 351

Text #12

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8102. Min Act: -0.0000

Data Index: 969016 (C4 (Web Text))

Max Activating Token Index: 305

Text #13

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.9119. Min Act: -0.0000

Data Index: 538005 (C4 (Web Text))

Max Activating Token Index: 355

Text #14

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8924. Min Act: -0.0000

Data Index: 1172281 (C4 (Web Text))

Max Activating Token Index: 319

Text #15

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8800. Min Act: -0.0000

Data Index: 120234 (C4 (Web Text))

Max Activating Token Index: 78

Text #16

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8652. Min Act: -0.0000

Data Index: 982113 (C4 (Web Text))

Max Activating Token Index: 200

Text #17

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.9559. Min Act: -0.0000

Data Index: 857928 (C4 (Web Text))

Max Activating Token Index: 407

Text #18

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8359. Min Act: -0.0000

Data Index: 777818 (C4 (Web Text))

Max Activating Token Index: 260

Text #19

Max Range: 2.6289. Min Range: -2.6289

Max Act: 1.8180. Min Act: -0.0000

Data Index: 694685 (C4 (Web Text))

Max Activating Token Index: 621