Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3056 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1912. Min Act: -0.0000

Data Index: 1334510 (C4 (Web Text))

Max Activating Token Index: 729

Text #1

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1629. Min Act: -0.0000

Data Index: 110680 (C4 (Web Text))

Max Activating Token Index: 919

Text #2

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1353. Min Act: -0.0000

Data Index: 324323 (C4 (Web Text))

Max Activating Token Index: 840

Text #3

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1266. Min Act: -0.0000

Data Index: 234311 (C4 (Web Text))

Max Activating Token Index: 559

Text #4

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1204. Min Act: -0.0000

Data Index: 131231 (C4 (Web Text))

Max Activating Token Index: 147

Text #5

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1139. Min Act: -0.0000

Data Index: 698483 (C4 (Web Text))

Max Activating Token Index: 946

Text #6

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1142. Min Act: -0.0000

Data Index: 1239149 (C4 (Web Text))

Max Activating Token Index: 445

Text #7

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1066. Min Act: -0.0000

Data Index: 670670 (C4 (Web Text))

Max Activating Token Index: 267

Text #8

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0993. Min Act: -0.0000

Data Index: 1163623 (C4 (Web Text))

Max Activating Token Index: 494

Text #9

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1023. Min Act: -0.0000

Data Index: 665173 (C4 (Web Text))

Max Activating Token Index: 788

Text #10

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1007. Min Act: -0.0000

Data Index: 1150940 (C4 (Web Text))

Max Activating Token Index: 59

Text #11

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0969. Min Act: -0.0000

Data Index: 626200 (C4 (Web Text))

Max Activating Token Index: 680

Text #12

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0974. Min Act: -0.0000

Data Index: 563489 (C4 (Web Text))

Max Activating Token Index: 142

Text #13

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.1010. Min Act: -0.0000

Data Index: 725559 (C4 (Web Text))

Max Activating Token Index: 531

Text #14

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0849. Min Act: -0.0000

Data Index: 664670 (C4 (Web Text))

Max Activating Token Index: 697

Text #15

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0967. Min Act: -0.0000

Data Index: 196448 (C4 (Web Text))

Max Activating Token Index: 624

Text #16

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0914. Min Act: -0.0000

Data Index: 673745 (C4 (Web Text))

Max Activating Token Index: 279

Text #17

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0867. Min Act: -0.0001

Data Index: 974273 (C4 (Web Text))

Max Activating Token Index: 903

Text #18

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0828. Min Act: -0.0000

Data Index: 1209822 (C4 (Web Text))

Max Activating Token Index: 831

Text #19

Max Range: 0.1912. Min Range: -0.1912

Max Act: 0.0846. Min Act: -0.0000

Data Index: 1257280 (C4 (Web Text))

Max Activating Token Index: 259