Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1942 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.6896. Min Act: -0.0000

Data Index: 1323427 (C4 (Web Text))

Max Activating Token Index: 925

Text #1

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.3999. Min Act: -0.0000

Data Index: 298388 (C4 (Web Text))

Max Activating Token Index: 930

Text #2

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.4483. Min Act: -0.0000

Data Index: 229162 (C4 (Web Text))

Max Activating Token Index: 660

Text #3

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.4717. Min Act: -0.0000

Data Index: 1266635 (C4 (Web Text))

Max Activating Token Index: 785

Text #4

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.7034. Min Act: -0.0000

Data Index: 601670 (C4 (Web Text))

Max Activating Token Index: 856

Text #5

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.4212. Min Act: -0.0000

Data Index: 1349108 (C4 (Web Text))

Max Activating Token Index: 974

Text #6

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.3858. Min Act: -0.0000

Data Index: 1102376 (C4 (Web Text))

Max Activating Token Index: 224

Text #7

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.3431. Min Act: -0.0000

Data Index: 1058636 (C4 (Web Text))

Max Activating Token Index: 312

Text #8

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.3548. Min Act: -0.0000

Data Index: 343539 (C4 (Web Text))

Max Activating Token Index: 678

Text #9

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.2861. Min Act: -0.0000

Data Index: 1348865 (C4 (Web Text))

Max Activating Token Index: 193

Text #10

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.3338. Min Act: -0.0000

Data Index: 374676 (C4 (Web Text))

Max Activating Token Index: 640

Text #11

Max Range: 2.7034. Min Range: -2.7034

Max Act: 2.3333. Min Act: -0.0000

Data Index: 558182 (C4 (Web Text))

Max Activating Token Index: 115

Text #12

Max Range: 2.7034. Min Range: -2.7034

Max Act: 1.1765. Min Act: -0.0000

Data Index: 883174 (C4 (Web Text))

Max Activating Token Index: 31

Text #13

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.8207. Min Act: -0.0000

Data Index: 262918 (C4 (Web Text))

Max Activating Token Index: 206

Text #14

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.8722. Min Act: -0.0000

Data Index: 1274694 (C4 (Web Text))

Max Activating Token Index: 317

Text #15

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.8747. Min Act: -0.0000

Data Index: 528755 (C4 (Web Text))

Max Activating Token Index: 367

Text #16

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.7694. Min Act: -0.0000

Data Index: 468162 (C4 (Web Text))

Max Activating Token Index: 644

Text #17

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.7486. Min Act: -0.0000

Data Index: 757595 (C4 (Web Text))

Max Activating Token Index: 907

Text #18

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.6860. Min Act: -0.0000

Data Index: 212505 (C4 (Web Text))

Max Activating Token Index: 662

Text #19

Max Range: 2.7034. Min Range: -2.7034

Max Act: 0.7782. Min Act: -0.0000

Data Index: 514355 (C4 (Web Text))

Max Activating Token Index: 568