Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4208 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0014. Min Range: -2.0014

Max Act: 2.0014. Min Act: -0.0000

Data Index: 423442 (C4 (Web Text))

Max Activating Token Index: 918

Text #1

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.9307. Min Act: -0.0000

Data Index: 1015827 (C4 (Web Text))

Max Activating Token Index: 747

Text #2

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8255. Min Act: -0.0000

Data Index: 516817 (C4 (Web Text))

Max Activating Token Index: 474

Text #3

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8707. Min Act: -0.0000

Data Index: 268342 (C4 (Web Text))

Max Activating Token Index: 496

Text #4

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.9291. Min Act: -0.0000

Data Index: 1053388 (C4 (Web Text))

Max Activating Token Index: 20

Text #5

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8160. Min Act: -0.0000

Data Index: 924190 (C4 (Web Text))

Max Activating Token Index: 557

Text #6

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8993. Min Act: -0.0000

Data Index: 572733 (C4 (Web Text))

Max Activating Token Index: 115

Text #7

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8486. Min Act: -0.0000

Data Index: 797026 (C4 (Web Text))

Max Activating Token Index: 892

Text #8

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8443. Min Act: -0.0000

Data Index: 972882 (C4 (Web Text))

Max Activating Token Index: 344

Text #9

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.6228. Min Act: -0.0000

Data Index: 634852 (C4 (Web Text))

Max Activating Token Index: 803

Text #10

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.8808. Min Act: -0.0000

Data Index: 1003568 (C4 (Web Text))

Max Activating Token Index: 801

Text #11

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.6485. Min Act: -0.0000

Data Index: 884351 (C4 (Web Text))

Max Activating Token Index: 572

Text #12

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.5323. Min Act: -0.0000

Data Index: 21591 (C4 (Web Text))

Max Activating Token Index: 249

Text #13

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.6946. Min Act: -0.0000

Data Index: 316357 (C4 (Web Text))

Max Activating Token Index: 736

Text #14

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.5854. Min Act: -0.0000

Data Index: 871632 (C4 (Web Text))

Max Activating Token Index: 395

Text #15

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.5438. Min Act: -0.0000

Data Index: 86827 (C4 (Web Text))

Max Activating Token Index: 250

Text #16

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.4561. Min Act: -0.0000

Data Index: 219317 (C4 (Web Text))

Max Activating Token Index: 457

Text #17

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.4228. Min Act: -0.0000

Data Index: 382475 (C4 (Web Text))

Max Activating Token Index: 978

Text #18

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.5008. Min Act: -0.0000

Data Index: 1343132 (C4 (Web Text))

Max Activating Token Index: 605

Text #19

Max Range: 2.0014. Min Range: -2.0014

Max Act: 1.4686. Min Act: -0.0000

Data Index: 149528 (C4 (Web Text))

Max Activating Token Index: 599