Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4963 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2379. Min Range: -1.2379

Max Act: 1.2379. Min Act: -0.0000

Data Index: 848714 (C4 (Web Text))

Max Activating Token Index: 58

Text #1

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.8893. Min Act: -0.0000

Data Index: 1053610 (C4 (Web Text))

Max Activating Token Index: 335

Text #2

Max Range: 1.2379. Min Range: -1.2379

Max Act: 1.0035. Min Act: -0.0000

Data Index: 66631 (C4 (Web Text))

Max Activating Token Index: 341

Text #3

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.9150. Min Act: -0.0000

Data Index: 625333 (C4 (Web Text))

Max Activating Token Index: 640

Text #4

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.9059. Min Act: -0.0000

Data Index: 1314910 (C4 (Web Text))

Max Activating Token Index: 302

Text #5

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.8289. Min Act: -0.0000

Data Index: 211133 (C4 (Web Text))

Max Activating Token Index: 113

Text #6

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7957. Min Act: -0.0000

Data Index: 761140 (C4 (Web Text))

Max Activating Token Index: 213

Text #7

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.8143. Min Act: -0.0000

Data Index: 638129 (C4 (Web Text))

Max Activating Token Index: 677

Text #8

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.6449. Min Act: -0.0000

Data Index: 96757 (C4 (Web Text))

Max Activating Token Index: 685

Text #9

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.8156. Min Act: -0.0000

Data Index: 941337 (C4 (Web Text))

Max Activating Token Index: 518

Text #10

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7922. Min Act: -0.0000

Data Index: 360023 (C4 (Web Text))

Max Activating Token Index: 156

Text #11

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7810. Min Act: -0.0000

Data Index: 1039164 (C4 (Web Text))

Max Activating Token Index: 604

Text #12

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7946. Min Act: -0.0000

Data Index: 931280 (C4 (Web Text))

Max Activating Token Index: 345

Text #13

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7435. Min Act: 0.0000

Data Index: 732341 (C4 (Web Text))

Max Activating Token Index: 607

Text #14

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7539. Min Act: -0.0000

Data Index: 364664 (C4 (Web Text))

Max Activating Token Index: 531

Text #15

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7985. Min Act: -0.0000

Data Index: 973893 (C4 (Web Text))

Max Activating Token Index: 55

Text #16

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7078. Min Act: -0.0000

Data Index: 207834 (C4 (Web Text))

Max Activating Token Index: 652

Text #17

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7498. Min Act: -0.0000

Data Index: 430310 (C4 (Web Text))

Max Activating Token Index: 431

Text #18

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7236. Min Act: -0.0000

Data Index: 748053 (C4 (Web Text))

Max Activating Token Index: 711

Text #19

Max Range: 1.2379. Min Range: -1.2379

Max Act: 0.7982. Min Act: -0.0000

Data Index: 1041431 (C4 (Web Text))

Max Activating Token Index: 704