Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2746 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1538. Min Act: -0.0000

Data Index: 679632 (C4 (Web Text))

Max Activating Token Index: 471

Text #1

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1349. Min Act: -0.0000

Data Index: 940707 (C4 (Web Text))

Max Activating Token Index: 787

Text #2

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1312. Min Act: -0.0000

Data Index: 989762 (C4 (Web Text))

Max Activating Token Index: 465

Text #3

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1257. Min Act: -0.0000

Data Index: 808899 (C4 (Web Text))

Max Activating Token Index: 779

Text #4

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1246. Min Act: -0.0000

Data Index: 741198 (C4 (Web Text))

Max Activating Token Index: 54

Text #5

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1244. Min Act: -0.0000

Data Index: 1430584 (Python Code)

Max Activating Token Index: 522

Text #6

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1219. Min Act: -0.0000

Data Index: 274528 (C4 (Web Text))

Max Activating Token Index: 396

Text #7

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1087. Min Act: -0.0000

Data Index: 4236 (C4 (Web Text))

Max Activating Token Index: 750

Text #8

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1141. Min Act: -0.0000

Data Index: 961347 (C4 (Web Text))

Max Activating Token Index: 962

Text #9

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1134. Min Act: -0.0000

Data Index: 1234565 (C4 (Web Text))

Max Activating Token Index: 228

Text #10

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1119. Min Act: -0.0000

Data Index: 1297311 (C4 (Web Text))

Max Activating Token Index: 672

Text #11

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1106. Min Act: -0.0000

Data Index: 123318 (C4 (Web Text))

Max Activating Token Index: 709

Text #12

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1066. Min Act: -0.0000

Data Index: 1190982 (C4 (Web Text))

Max Activating Token Index: 871

Text #13

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1102. Min Act: -0.0000

Data Index: 258442 (C4 (Web Text))

Max Activating Token Index: 299

Text #14

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1090. Min Act: -0.0000

Data Index: 345175 (C4 (Web Text))

Max Activating Token Index: 533

Text #15

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1006. Min Act: -0.0000

Data Index: 608604 (C4 (Web Text))

Max Activating Token Index: 920

Text #16

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1033. Min Act: -0.0000

Data Index: 1584791 (Python Code)

Max Activating Token Index: 492

Text #17

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.0958. Min Act: -0.0000

Data Index: 80378 (C4 (Web Text))

Max Activating Token Index: 448

Text #18

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.1012. Min Act: -0.0000

Data Index: 933157 (C4 (Web Text))

Max Activating Token Index: 332

Text #19

Max Range: 0.1538. Min Range: -0.1538

Max Act: 0.0987. Min Act: -0.0000

Data Index: 309405 (C4 (Web Text))

Max Activating Token Index: 397