Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2881 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.9465. Min Act: -0.0000

Data Index: 213426 (C4 (Web Text))

Max Activating Token Index: 846

Text #1

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.9435. Min Act: -0.0000

Data Index: 1180480 (C4 (Web Text))

Max Activating Token Index: 683

Text #2

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8122. Min Act: -0.0000

Data Index: 1310299 (C4 (Web Text))

Max Activating Token Index: 762

Text #3

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.9193. Min Act: -0.0000

Data Index: 1229588 (C4 (Web Text))

Max Activating Token Index: 861

Text #4

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.9058. Min Act: -0.0000

Data Index: 26957 (C4 (Web Text))

Max Activating Token Index: 216

Text #5

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8984. Min Act: -0.0000

Data Index: 722411 (C4 (Web Text))

Max Activating Token Index: 128

Text #6

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8970. Min Act: -0.0000

Data Index: 830371 (C4 (Web Text))

Max Activating Token Index: 156

Text #7

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8141. Min Act: -0.0000

Data Index: 715247 (C4 (Web Text))

Max Activating Token Index: 926

Text #8

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8618. Min Act: -0.0000

Data Index: 833858 (C4 (Web Text))

Max Activating Token Index: 855

Text #9

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8105. Min Act: -0.0000

Data Index: 736389 (C4 (Web Text))

Max Activating Token Index: 911

Text #10

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8355. Min Act: -0.0000

Data Index: 183864 (C4 (Web Text))

Max Activating Token Index: 843

Text #11

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8394. Min Act: -0.0000

Data Index: 790755 (C4 (Web Text))

Max Activating Token Index: 1003

Text #12

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.8131. Min Act: -0.0000

Data Index: 997689 (C4 (Web Text))

Max Activating Token Index: 494

Text #13

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7991. Min Act: -0.0000

Data Index: 1132953 (C4 (Web Text))

Max Activating Token Index: 438

Text #14

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7624. Min Act: -0.0000

Data Index: 1243514 (C4 (Web Text))

Max Activating Token Index: 305

Text #15

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7858. Min Act: -0.0000

Data Index: 263201 (C4 (Web Text))

Max Activating Token Index: 449

Text #16

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7500. Min Act: -0.0000

Data Index: 1224663 (C4 (Web Text))

Max Activating Token Index: 174

Text #17

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7354. Min Act: -0.0000

Data Index: 1100286 (C4 (Web Text))

Max Activating Token Index: 525

Text #18

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7323. Min Act: -0.0000

Data Index: 1098012 (C4 (Web Text))

Max Activating Token Index: 846

Text #19

Max Range: 0.9465. Min Range: -0.9465

Max Act: 0.7283. Min Act: -0.0000

Data Index: 359637 (C4 (Web Text))

Max Activating Token Index: 50