Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2006 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.8923. Min Act: -0.0000

Data Index: 599251 (The Pile)

Max Activating Token Index: 795

Text #1

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.7493. Min Act: -0.0000

Data Index: 1699410 (The Pile)

Max Activating Token Index: 520

Text #2

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.5228. Min Act: -0.0000

Data Index: 707661 (The Pile)

Max Activating Token Index: 156

Text #3

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.5731. Min Act: -0.0000

Data Index: 799064 (The Pile)

Max Activating Token Index: 929

Text #4

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.3765. Min Act: -0.0000

Data Index: 1879260 (The Pile)

Max Activating Token Index: 891

Text #5

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.5283. Min Act: -0.0000

Data Index: 1517754 (The Pile)

Max Activating Token Index: 91

Text #6

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4215. Min Act: -0.0000

Data Index: 451610 (The Pile)

Max Activating Token Index: 813

Text #7

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.5675. Min Act: -0.0000

Data Index: 1976780 (The Pile)

Max Activating Token Index: 821

Text #8

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4066. Min Act: -0.0000

Data Index: 1923118 (The Pile)

Max Activating Token Index: 964

Text #9

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4305. Min Act: -0.0000

Data Index: 211328 (The Pile)

Max Activating Token Index: 788

Text #10

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4436. Min Act: -0.0000

Data Index: 1708272 (The Pile)

Max Activating Token Index: 402

Text #11

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4458. Min Act: -0.0000

Data Index: 427208 (The Pile)

Max Activating Token Index: 415

Text #12

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.3888. Min Act: -0.0000

Data Index: 419919 (The Pile)

Max Activating Token Index: 970

Text #13

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.3406. Min Act: -0.0000

Data Index: 1383529 (The Pile)

Max Activating Token Index: 1016

Text #14

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.3758. Min Act: -0.0000

Data Index: 294923 (The Pile)

Max Activating Token Index: 564

Text #15

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.3158. Min Act: -0.0000

Data Index: 236353 (The Pile)

Max Activating Token Index: 630

Text #16

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4176. Min Act: -0.0000

Data Index: 1255163 (The Pile)

Max Activating Token Index: 78

Text #17

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4872. Min Act: -0.0000

Data Index: 173480 (The Pile)

Max Activating Token Index: 87

Text #18

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.3901. Min Act: -0.0000

Data Index: 575296 (The Pile)

Max Activating Token Index: 242

Text #19

Max Range: 1.8923. Min Range: -1.8923

Max Act: 1.4253. Min Act: -0.0000

Data Index: 28209 (The Pile)

Max Activating Token Index: 321