Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4233 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.4095. Min Act: -0.0000

Data Index: 977004 (C4 (Web Text))

Max Activating Token Index: 1006

Text #1

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.1518. Min Act: -0.0000

Data Index: 1247322 (C4 (Web Text))

Max Activating Token Index: 861

Text #2

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.6135. Min Act: -0.0001

Data Index: 477202 (C4 (Web Text))

Max Activating Token Index: 128

Text #3

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.2002. Min Act: -0.0001

Data Index: 873079 (C4 (Web Text))

Max Activating Token Index: 1020

Text #4

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.2870. Min Act: -0.0000

Data Index: 667240 (C4 (Web Text))

Max Activating Token Index: 5

Text #5

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.0989. Min Act: -0.0000

Data Index: 10595 (C4 (Web Text))

Max Activating Token Index: 145

Text #6

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.3584. Min Act: -0.0000

Data Index: 783120 (C4 (Web Text))

Max Activating Token Index: 823

Text #7

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.1593. Min Act: -0.0000

Data Index: 1138210 (C4 (Web Text))

Max Activating Token Index: 467

Text #8

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.3418. Min Act: -0.0000

Data Index: 611363 (C4 (Web Text))

Max Activating Token Index: 512

Text #9

Max Range: 6.6135. Min Range: -6.6135

Max Act: 5.9772. Min Act: -0.0000

Data Index: 16294 (C4 (Web Text))

Max Activating Token Index: 195

Text #10

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.1542. Min Act: -0.0000

Data Index: 1167616 (C4 (Web Text))

Max Activating Token Index: 41

Text #11

Max Range: 6.6135. Min Range: -6.6135

Max Act: 5.9676. Min Act: -0.0000

Data Index: 698315 (C4 (Web Text))

Max Activating Token Index: 470

Text #12

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.4845. Min Act: -0.0000

Data Index: 1179565 (C4 (Web Text))

Max Activating Token Index: 12

Text #13

Max Range: 6.6135. Min Range: -6.6135

Max Act: 5.9403. Min Act: -0.0000

Data Index: 314678 (C4 (Web Text))

Max Activating Token Index: 866

Text #14

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.1385. Min Act: -0.0000

Data Index: 594911 (C4 (Web Text))

Max Activating Token Index: 861

Text #15

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.0347. Min Act: -0.0000

Data Index: 535257 (C4 (Web Text))

Max Activating Token Index: 42

Text #16

Max Range: 6.6135. Min Range: -6.6135

Max Act: 6.0207. Min Act: -0.0000

Data Index: 541881 (C4 (Web Text))

Max Activating Token Index: 951

Text #17

Max Range: 6.6135. Min Range: -6.6135

Max Act: 5.9416. Min Act: -0.0000

Data Index: 42311 (C4 (Web Text))

Max Activating Token Index: 10

Text #18

Max Range: 6.6135. Min Range: -6.6135

Max Act: 5.5166. Min Act: -0.0000

Data Index: 715731 (C4 (Web Text))

Max Activating Token Index: 865

Text #19

Max Range: 6.6135. Min Range: -6.6135

Max Act: 5.7174. Min Act: -0.0000

Data Index: 1149003 (C4 (Web Text))

Max Activating Token Index: 874