Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3002 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2094. Min Range: -3.2094

Max Act: 3.2094. Min Act: -0.0000

Data Index: 307864 (C4 (Web Text))

Max Activating Token Index: 33

Text #1

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.6887. Min Act: -0.0000

Data Index: 81102 (C4 (Web Text))

Max Activating Token Index: 705

Text #2

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.7728. Min Act: -0.0000

Data Index: 787904 (C4 (Web Text))

Max Activating Token Index: 824

Text #3

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.7647. Min Act: -0.0000

Data Index: 1086875 (C4 (Web Text))

Max Activating Token Index: 942

Text #4

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.7563. Min Act: -0.0000

Data Index: 286117 (C4 (Web Text))

Max Activating Token Index: 323

Text #5

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.6191. Min Act: -0.0000

Data Index: 1012035 (C4 (Web Text))

Max Activating Token Index: 824

Text #6

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3391. Min Act: -0.0000

Data Index: 552808 (C4 (Web Text))

Max Activating Token Index: 880

Text #7

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.5162. Min Act: -0.0000

Data Index: 629721 (C4 (Web Text))

Max Activating Token Index: 32

Text #8

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.5044. Min Act: -0.0000

Data Index: 1176741 (C4 (Web Text))

Max Activating Token Index: 952

Text #9

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3221. Min Act: -0.0000

Data Index: 974776 (C4 (Web Text))

Max Activating Token Index: 978

Text #10

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.1834. Min Act: -0.0000

Data Index: 1307866 (C4 (Web Text))

Max Activating Token Index: 881

Text #11

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.5689. Min Act: -0.0000

Data Index: 743026 (C4 (Web Text))

Max Activating Token Index: 355

Text #12

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3669. Min Act: -0.0000

Data Index: 1275748 (C4 (Web Text))

Max Activating Token Index: 935

Text #13

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3209. Min Act: -0.0000

Data Index: 31771 (C4 (Web Text))

Max Activating Token Index: 990

Text #14

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.4687. Min Act: -0.0000

Data Index: 995509 (C4 (Web Text))

Max Activating Token Index: 61

Text #15

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3360. Min Act: -0.0000

Data Index: 499214 (C4 (Web Text))

Max Activating Token Index: 250

Text #16

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.2147. Min Act: -0.0000

Data Index: 360553 (C4 (Web Text))

Max Activating Token Index: 657

Text #17

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3284. Min Act: -0.0000

Data Index: 846319 (C4 (Web Text))

Max Activating Token Index: 262

Text #18

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.3319. Min Act: -0.0000

Data Index: 37766 (C4 (Web Text))

Max Activating Token Index: 245

Text #19

Max Range: 3.2094. Min Range: -3.2094

Max Act: 2.2786. Min Act: -0.0000

Data Index: 565949 (C4 (Web Text))

Max Activating Token Index: 928