Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2357 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.6566. Min Act: -0.0000

Data Index: 1268839 (C4 (Web Text))

Max Activating Token Index: 1019

Text #1

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.6360. Min Act: -0.0000

Data Index: 25928 (C4 (Web Text))

Max Activating Token Index: 292

Text #2

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.1978. Min Act: -0.0000

Data Index: 1090247 (C4 (Web Text))

Max Activating Token Index: 228

Text #3

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.4246. Min Act: -0.0000

Data Index: 784416 (C4 (Web Text))

Max Activating Token Index: 578

Text #4

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.2728. Min Act: -0.0000

Data Index: 1300375 (C4 (Web Text))

Max Activating Token Index: 979

Text #5

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.2717. Min Act: -0.0000

Data Index: 536757 (C4 (Web Text))

Max Activating Token Index: 1012

Text #6

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.2474. Min Act: -0.0000

Data Index: 1112639 (C4 (Web Text))

Max Activating Token Index: 51

Text #7

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.1977. Min Act: -0.0000

Data Index: 525067 (C4 (Web Text))

Max Activating Token Index: 182

Text #8

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.2474. Min Act: -0.0000

Data Index: 553262 (C4 (Web Text))

Max Activating Token Index: 907

Text #9

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.1284. Min Act: -0.0000

Data Index: 981403 (C4 (Web Text))

Max Activating Token Index: 1019

Text #10

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.1169. Min Act: -0.0000

Data Index: 271432 (C4 (Web Text))

Max Activating Token Index: 588

Text #11

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.3107. Min Act: -0.0000

Data Index: 223011 (C4 (Web Text))

Max Activating Token Index: 905

Text #12

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.8489. Min Act: -0.0000

Data Index: 166258 (C4 (Web Text))

Max Activating Token Index: 853

Text #13

Max Range: 2.6566. Min Range: -2.6566

Max Act: 2.0231. Min Act: -0.0000

Data Index: 1128709 (C4 (Web Text))

Max Activating Token Index: 17

Text #14

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.9187. Min Act: -0.0000

Data Index: 1229530 (C4 (Web Text))

Max Activating Token Index: 1018

Text #15

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.9750. Min Act: -0.0000

Data Index: 1261545 (C4 (Web Text))

Max Activating Token Index: 72

Text #16

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.9697. Min Act: -0.0000

Data Index: 543586 (C4 (Web Text))

Max Activating Token Index: 723

Text #17

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.8366. Min Act: -0.0000

Data Index: 439096 (C4 (Web Text))

Max Activating Token Index: 936

Text #18

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.9337. Min Act: -0.0000

Data Index: 1627450 (Python Code)

Max Activating Token Index: 769

Text #19

Max Range: 2.6566. Min Range: -2.6566

Max Act: 1.9332. Min Act: -0.0000

Data Index: 964639 (C4 (Web Text))

Max Activating Token Index: 23