Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5425 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.6366. Min Act: -0.0000

Data Index: 973320 (C4 (Web Text))

Max Activating Token Index: 624

Text #1

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.5847. Min Act: -0.0000

Data Index: 109316 (C4 (Web Text))

Max Activating Token Index: 381

Text #2

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.4351. Min Act: -0.0000

Data Index: 630660 (C4 (Web Text))

Max Activating Token Index: 708

Text #3

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.1186. Min Act: -0.0000

Data Index: 870883 (C4 (Web Text))

Max Activating Token Index: 311

Text #4

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.0410. Min Act: -0.0000

Data Index: 1087742 (C4 (Web Text))

Max Activating Token Index: 269

Text #5

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.0022. Min Act: -0.0000

Data Index: 241653 (C4 (Web Text))

Max Activating Token Index: 252

Text #6

Max Range: 1.6366. Min Range: -1.6366

Max Act: 1.2682. Min Act: -0.0000

Data Index: 1186251 (C4 (Web Text))

Max Activating Token Index: 739

Text #7

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.9095. Min Act: -0.0000

Data Index: 734173 (C4 (Web Text))

Max Activating Token Index: 425

Text #8

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.8638. Min Act: -0.0000

Data Index: 583513 (C4 (Web Text))

Max Activating Token Index: 1002

Text #9

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.8724. Min Act: -0.0000

Data Index: 1332431 (C4 (Web Text))

Max Activating Token Index: 538

Text #10

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.8488. Min Act: -0.0000

Data Index: 1115140 (C4 (Web Text))

Max Activating Token Index: 328

Text #11

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.8502. Min Act: -0.0000

Data Index: 711243 (C4 (Web Text))

Max Activating Token Index: 761

Text #12

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.8422. Min Act: -0.0000

Data Index: 717367 (C4 (Web Text))

Max Activating Token Index: 487

Text #13

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.8374. Min Act: -0.0000

Data Index: 51860 (C4 (Web Text))

Max Activating Token Index: 378

Text #14

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.7462. Min Act: -0.0000

Data Index: 89694 (C4 (Web Text))

Max Activating Token Index: 87

Text #15

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.7581. Min Act: -0.0000

Data Index: 218537 (C4 (Web Text))

Max Activating Token Index: 866

Text #16

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.7141. Min Act: -0.0000

Data Index: 886953 (C4 (Web Text))

Max Activating Token Index: 37

Text #17

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.7459. Min Act: -0.0000

Data Index: 1052590 (C4 (Web Text))

Max Activating Token Index: 635

Text #18

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.7750. Min Act: -0.0000

Data Index: 1151410 (C4 (Web Text))

Max Activating Token Index: 197

Text #19

Max Range: 1.6366. Min Range: -1.6366

Max Act: 0.7952. Min Act: -0.0000

Data Index: 980760 (C4 (Web Text))

Max Activating Token Index: 170