Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5236 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.3153. Min Act: -0.0000

Data Index: 108888 (C4 (Web Text))

Max Activating Token Index: 966

Text #1

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2805. Min Act: -0.0001

Data Index: 13743 (C4 (Web Text))

Max Activating Token Index: 48

Text #2

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2370. Min Act: -0.0000

Data Index: 370405 (C4 (Web Text))

Max Activating Token Index: 856

Text #3

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2558. Min Act: -0.0000

Data Index: 530958 (C4 (Web Text))

Max Activating Token Index: 805

Text #4

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2602. Min Act: -0.0000

Data Index: 1152434 (C4 (Web Text))

Max Activating Token Index: 563

Text #5

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2568. Min Act: -0.0000

Data Index: 202046 (C4 (Web Text))

Max Activating Token Index: 151

Text #6

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2871. Min Act: -0.0000

Data Index: 1209535 (C4 (Web Text))

Max Activating Token Index: 775

Text #7

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2560. Min Act: -0.0000

Data Index: 1086294 (C4 (Web Text))

Max Activating Token Index: 835

Text #8

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.1914. Min Act: -0.0000

Data Index: 738183 (C4 (Web Text))

Max Activating Token Index: 186

Text #9

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2554. Min Act: -0.0000

Data Index: 930923 (C4 (Web Text))

Max Activating Token Index: 133

Text #10

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2432. Min Act: -0.0000

Data Index: 802408 (C4 (Web Text))

Max Activating Token Index: 871

Text #11

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2440. Min Act: -0.0000

Data Index: 409396 (C4 (Web Text))

Max Activating Token Index: 70

Text #12

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2443. Min Act: -0.0000

Data Index: 84652 (C4 (Web Text))

Max Activating Token Index: 844

Text #13

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2321. Min Act: -0.0000

Data Index: 917473 (C4 (Web Text))

Max Activating Token Index: 510

Text #14

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2397. Min Act: -0.0000

Data Index: 1139438 (C4 (Web Text))

Max Activating Token Index: 139

Text #15

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2293. Min Act: -0.0000

Data Index: 887891 (C4 (Web Text))

Max Activating Token Index: 967

Text #16

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2358. Min Act: -0.0000

Data Index: 1250920 (C4 (Web Text))

Max Activating Token Index: 616

Text #17

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2141. Min Act: -0.0001

Data Index: 1038495 (C4 (Web Text))

Max Activating Token Index: 815

Text #18

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2137. Min Act: -0.0000

Data Index: 201133 (C4 (Web Text))

Max Activating Token Index: 787

Text #19

Max Range: 0.3153. Min Range: -0.3153

Max Act: 0.2433. Min Act: -0.0000

Data Index: 227100 (C4 (Web Text))

Max Activating Token Index: 594