Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2656 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.3209. Min Act: -0.0000

Data Index: 77448 (C4 (Web Text))

Max Activating Token Index: 801

Text #1

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.3127. Min Act: -0.0000

Data Index: 1005091 (C4 (Web Text))

Max Activating Token Index: 236

Text #2

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2975. Min Act: -0.0000

Data Index: 941428 (C4 (Web Text))

Max Activating Token Index: 618

Text #3

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2820. Min Act: -0.0000

Data Index: 396568 (C4 (Web Text))

Max Activating Token Index: 801

Text #4

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2588. Min Act: -0.0000

Data Index: 445616 (C4 (Web Text))

Max Activating Token Index: 72

Text #5

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2443. Min Act: -0.0000

Data Index: 454480 (C4 (Web Text))

Max Activating Token Index: 1015

Text #6

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2417. Min Act: -0.0000

Data Index: 115482 (C4 (Web Text))

Max Activating Token Index: 935

Text #7

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2272. Min Act: -0.0000

Data Index: 1008050 (C4 (Web Text))

Max Activating Token Index: 855

Text #8

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2212. Min Act: -0.0000

Data Index: 606206 (C4 (Web Text))

Max Activating Token Index: 338

Text #9

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2319. Min Act: -0.0000

Data Index: 1233942 (C4 (Web Text))

Max Activating Token Index: 563

Text #10

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2150. Min Act: -0.0000

Data Index: 860970 (C4 (Web Text))

Max Activating Token Index: 1022

Text #11

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2144. Min Act: -0.0000

Data Index: 748077 (C4 (Web Text))

Max Activating Token Index: 340

Text #12

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2052. Min Act: -0.0000

Data Index: 1309530 (C4 (Web Text))

Max Activating Token Index: 753

Text #13

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2271. Min Act: -0.0000

Data Index: 155959 (C4 (Web Text))

Max Activating Token Index: 842

Text #14

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2272. Min Act: -0.0000

Data Index: 83908 (C4 (Web Text))

Max Activating Token Index: 584

Text #15

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2277. Min Act: 0.0000

Data Index: 885185 (C4 (Web Text))

Max Activating Token Index: 246

Text #16

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2184. Min Act: 0.0000

Data Index: 447156 (C4 (Web Text))

Max Activating Token Index: 218

Text #17

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2176. Min Act: -0.0000

Data Index: 642963 (C4 (Web Text))

Max Activating Token Index: 255

Text #18

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2154. Min Act: 0.0000

Data Index: 258858 (C4 (Web Text))

Max Activating Token Index: 67

Text #19

Max Range: 0.3209. Min Range: -0.3209

Max Act: 0.2043. Min Act: -0.0000

Data Index: 1088913 (C4 (Web Text))

Max Activating Token Index: 351