Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2461 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.5182. Min Act: -0.0001

Data Index: 1358502 (C4 (Web Text))

Max Activating Token Index: 17

Text #1

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.3475. Min Act: -0.0001

Data Index: 231289 (C4 (Web Text))

Max Activating Token Index: 4

Text #2

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0138. Min Act: -0.0002

Data Index: 697141 (C4 (Web Text))

Max Activating Token Index: 67

Text #3

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.1212. Min Act: -0.0002

Data Index: 1261237 (C4 (Web Text))

Max Activating Token Index: 71

Text #4

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0779. Min Act: -0.0001

Data Index: 681247 (C4 (Web Text))

Max Activating Token Index: 265

Text #5

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.8431. Min Act: -0.0002

Data Index: 48641 (C4 (Web Text))

Max Activating Token Index: 933

Text #6

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0660. Min Act: -0.0001

Data Index: 118926 (C4 (Web Text))

Max Activating Token Index: 743

Text #7

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0164. Min Act: -0.0001

Data Index: 740367 (C4 (Web Text))

Max Activating Token Index: 99

Text #8

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0320. Min Act: -0.0002

Data Index: 252801 (C4 (Web Text))

Max Activating Token Index: 454

Text #9

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0809. Min Act: -0.0002

Data Index: 956635 (C4 (Web Text))

Max Activating Token Index: 935

Text #10

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0065. Min Act: -0.0001

Data Index: 837610 (C4 (Web Text))

Max Activating Token Index: 118

Text #11

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.9858. Min Act: -0.0002

Data Index: 143644 (C4 (Web Text))

Max Activating Token Index: 1004

Text #12

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.9880. Min Act: -0.0002

Data Index: 1064373 (C4 (Web Text))

Max Activating Token Index: 21

Text #13

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.9265. Min Act: -0.0002

Data Index: 987010 (C4 (Web Text))

Max Activating Token Index: 72

Text #14

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0116. Min Act: -0.0002

Data Index: 738117 (C4 (Web Text))

Max Activating Token Index: 212

Text #15

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.9100. Min Act: -0.0002

Data Index: 29480 (C4 (Web Text))

Max Activating Token Index: 904

Text #16

Max Range: 1.5182. Min Range: -1.5182

Max Act: 1.0349. Min Act: -0.0001

Data Index: 162436 (C4 (Web Text))

Max Activating Token Index: 952

Text #17

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.8543. Min Act: -0.0002

Data Index: 1225882 (C4 (Web Text))

Max Activating Token Index: 720

Text #18

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.9362. Min Act: -0.0001

Data Index: 1547574 (Python Code)

Max Activating Token Index: 76

Text #19

Max Range: 1.5182. Min Range: -1.5182

Max Act: 0.8913. Min Act: -0.0002

Data Index: 1320758 (C4 (Web Text))

Max Activating Token Index: 877