Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5204 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.1252. Min Act: -0.0000

Data Index: 989009 (C4 (Web Text))

Max Activating Token Index: 594

Text #1

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.1015. Min Act: -0.0000

Data Index: 976531 (C4 (Web Text))

Max Activating Token Index: 806

Text #2

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.1249. Min Act: -0.0000

Data Index: 1222443 (C4 (Web Text))

Max Activating Token Index: 437

Text #3

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.1695. Min Act: -0.0000

Data Index: 417163 (C4 (Web Text))

Max Activating Token Index: 302

Text #4

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.0180. Min Act: -0.0000

Data Index: 62620 (C4 (Web Text))

Max Activating Token Index: 361

Text #5

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.0625. Min Act: -0.0000

Data Index: 534870 (C4 (Web Text))

Max Activating Token Index: 633

Text #6

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9907. Min Act: -0.0000

Data Index: 752598 (C4 (Web Text))

Max Activating Token Index: 323

Text #7

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.0318. Min Act: -0.0000

Data Index: 195046 (C4 (Web Text))

Max Activating Token Index: 298

Text #8

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9431. Min Act: -0.0000

Data Index: 406740 (C4 (Web Text))

Max Activating Token Index: 269

Text #9

Max Range: 1.1695. Min Range: -1.1695

Max Act: 1.0094. Min Act: -0.0000

Data Index: 916850 (C4 (Web Text))

Max Activating Token Index: 597

Text #10

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9923. Min Act: -0.0000

Data Index: 703389 (C4 (Web Text))

Max Activating Token Index: 467

Text #11

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9653. Min Act: -0.0000

Data Index: 1079554 (C4 (Web Text))

Max Activating Token Index: 583

Text #12

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9491. Min Act: -0.0000

Data Index: 1141861 (C4 (Web Text))

Max Activating Token Index: 481

Text #13

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9015. Min Act: -0.0000

Data Index: 1326345 (C4 (Web Text))

Max Activating Token Index: 972

Text #14

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.8423. Min Act: -0.0000

Data Index: 1351018 (C4 (Web Text))

Max Activating Token Index: 156

Text #15

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.8981. Min Act: -0.0000

Data Index: 1280572 (C4 (Web Text))

Max Activating Token Index: 947

Text #16

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.8267. Min Act: -0.0000

Data Index: 542841 (C4 (Web Text))

Max Activating Token Index: 1007

Text #17

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.8199. Min Act: -0.0000

Data Index: 1284853 (C4 (Web Text))

Max Activating Token Index: 292

Text #18

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.8215. Min Act: -0.0000

Data Index: 114625 (C4 (Web Text))

Max Activating Token Index: 856

Text #19

Max Range: 1.1695. Min Range: -1.1695

Max Act: 0.9109. Min Act: -0.0000

Data Index: 1344797 (C4 (Web Text))

Max Activating Token Index: 520