Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3760 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.6395. Min Act: -0.0000

Data Index: 825466 (C4 (Web Text))

Max Activating Token Index: 482

Text #1

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.2638. Min Act: -0.0000

Data Index: 825665 (C4 (Web Text))

Max Activating Token Index: 135

Text #2

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.2436. Min Act: -0.0000

Data Index: 767288 (C4 (Web Text))

Max Activating Token Index: 682

Text #3

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.1323. Min Act: -0.0000

Data Index: 962481 (C4 (Web Text))

Max Activating Token Index: 788

Text #4

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.1290. Min Act: -0.0000

Data Index: 1261634 (C4 (Web Text))

Max Activating Token Index: 287

Text #5

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.0949. Min Act: -0.0000

Data Index: 393976 (C4 (Web Text))

Max Activating Token Index: 943

Text #6

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.0548. Min Act: -0.0000

Data Index: 1085649 (C4 (Web Text))

Max Activating Token Index: 672

Text #7

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.0343. Min Act: -0.0000

Data Index: 62794 (C4 (Web Text))

Max Activating Token Index: 913

Text #8

Max Range: 1.6395. Min Range: -1.6395

Max Act: 1.0552. Min Act: -0.0000

Data Index: 1159140 (C4 (Web Text))

Max Activating Token Index: 544

Text #9

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.8872. Min Act: -0.0000

Data Index: 1242962 (C4 (Web Text))

Max Activating Token Index: 820

Text #10

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.8983. Min Act: -0.0000

Data Index: 945110 (C4 (Web Text))

Max Activating Token Index: 502

Text #11

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.9100. Min Act: -0.0000

Data Index: 994293 (C4 (Web Text))

Max Activating Token Index: 456

Text #12

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.8530. Min Act: -0.0000

Data Index: 973775 (C4 (Web Text))

Max Activating Token Index: 482

Text #13

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.8376. Min Act: -0.0000

Data Index: 934275 (C4 (Web Text))

Max Activating Token Index: 437

Text #14

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.7779. Min Act: -0.0000

Data Index: 1117397 (C4 (Web Text))

Max Activating Token Index: 881

Text #15

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.8184. Min Act: -0.0000

Data Index: 417798 (C4 (Web Text))

Max Activating Token Index: 865

Text #16

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.7744. Min Act: -0.0000

Data Index: 1142254 (C4 (Web Text))

Max Activating Token Index: 283

Text #17

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.7671. Min Act: -0.0000

Data Index: 935972 (C4 (Web Text))

Max Activating Token Index: 1017

Text #18

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.7904. Min Act: -0.0000

Data Index: 361428 (C4 (Web Text))

Max Activating Token Index: 13

Text #19

Max Range: 1.6395. Min Range: -1.6395

Max Act: 0.7609. Min Act: -0.0000

Data Index: 88685 (C4 (Web Text))

Max Activating Token Index: 785