Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1185 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.8192. Min Act: -0.0001

Data Index: 1358065 (C4 (Web Text))

Max Activating Token Index: 925

Text #1

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.8171. Min Act: -0.0001

Data Index: 527397 (C4 (Web Text))

Max Activating Token Index: 796

Text #2

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.7299. Min Act: -0.0000

Data Index: 648281 (C4 (Web Text))

Max Activating Token Index: 570

Text #3

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.6960. Min Act: -0.0000

Data Index: 425068 (C4 (Web Text))

Max Activating Token Index: 345

Text #4

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.6594. Min Act: -0.0001

Data Index: 1020841 (C4 (Web Text))

Max Activating Token Index: 931

Text #5

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.7482. Min Act: -0.0000

Data Index: 52159 (C4 (Web Text))

Max Activating Token Index: 216

Text #6

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.7186. Min Act: -0.0000

Data Index: 601210 (C4 (Web Text))

Max Activating Token Index: 355

Text #7

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.6477. Min Act: -0.0001

Data Index: 33203 (C4 (Web Text))

Max Activating Token Index: 836

Text #8

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.3741. Min Act: -0.0001

Data Index: 340937 (C4 (Web Text))

Max Activating Token Index: 625

Text #9

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.5946. Min Act: -0.0000

Data Index: 66501 (C4 (Web Text))

Max Activating Token Index: 191

Text #10

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.6407. Min Act: -0.0001

Data Index: 1271803 (C4 (Web Text))

Max Activating Token Index: 355

Text #11

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.6418. Min Act: -0.0001

Data Index: 378372 (C4 (Web Text))

Max Activating Token Index: 949

Text #12

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.3821. Min Act: -0.0001

Data Index: 1196546 (C4 (Web Text))

Max Activating Token Index: 675

Text #13

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.3724. Min Act: -0.0001

Data Index: 383804 (C4 (Web Text))

Max Activating Token Index: 422

Text #14

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.5277. Min Act: -0.0001

Data Index: 383069 (C4 (Web Text))

Max Activating Token Index: 236

Text #15

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.4365. Min Act: -0.0000

Data Index: 1031739 (C4 (Web Text))

Max Activating Token Index: 784

Text #16

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.4301. Min Act: -0.0001

Data Index: 1048297 (C4 (Web Text))

Max Activating Token Index: 339

Text #17

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.4382. Min Act: -0.0000

Data Index: 487157 (C4 (Web Text))

Max Activating Token Index: 26

Text #18

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.4332. Min Act: -0.0001

Data Index: 440017 (C4 (Web Text))

Max Activating Token Index: 122

Text #19

Max Range: 1.8192. Min Range: -1.8192

Max Act: 1.3648. Min Act: -0.0001

Data Index: 907382 (C4 (Web Text))

Max Activating Token Index: 716