Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 443 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.6185. Min Act: -0.0000

Data Index: 182625 (C4 (Web Text))

Max Activating Token Index: 447

Text #1

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.6368. Min Act: -0.0000

Data Index: 241487 (C4 (Web Text))

Max Activating Token Index: 33

Text #2

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5933. Min Act: -0.0000

Data Index: 219259 (C4 (Web Text))

Max Activating Token Index: 72

Text #3

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5491. Min Act: -0.0000

Data Index: 849848 (C4 (Web Text))

Max Activating Token Index: 82

Text #4

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5351. Min Act: -0.0000

Data Index: 478285 (C4 (Web Text))

Max Activating Token Index: 958

Text #5

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5345. Min Act: -0.0000

Data Index: 1263586 (C4 (Web Text))

Max Activating Token Index: 883

Text #6

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5153. Min Act: -0.0000

Data Index: 450631 (C4 (Web Text))

Max Activating Token Index: 189

Text #7

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5312. Min Act: -0.0000

Data Index: 1116647 (C4 (Web Text))

Max Activating Token Index: 888

Text #8

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5275. Min Act: -0.0000

Data Index: 282045 (C4 (Web Text))

Max Activating Token Index: 26

Text #9

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4958. Min Act: -0.0000

Data Index: 808741 (C4 (Web Text))

Max Activating Token Index: 514

Text #10

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.5006. Min Act: -0.0000

Data Index: 286166 (C4 (Web Text))

Max Activating Token Index: 26

Text #11

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4631. Min Act: -0.0000

Data Index: 962049 (C4 (Web Text))

Max Activating Token Index: 737

Text #12

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4884. Min Act: -0.0000

Data Index: 1174982 (C4 (Web Text))

Max Activating Token Index: 1003

Text #13

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4565. Min Act: -0.0001

Data Index: 1315957 (C4 (Web Text))

Max Activating Token Index: 204

Text #14

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4881. Min Act: -0.0000

Data Index: 1325942 (C4 (Web Text))

Max Activating Token Index: 124

Text #15

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4661. Min Act: -0.0001

Data Index: 1234030 (C4 (Web Text))

Max Activating Token Index: 57

Text #16

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4795. Min Act: -0.0000

Data Index: 855335 (C4 (Web Text))

Max Activating Token Index: 987

Text #17

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4953. Min Act: -0.0000

Data Index: 663414 (C4 (Web Text))

Max Activating Token Index: 893

Text #18

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4607. Min Act: -0.0001

Data Index: 595967 (C4 (Web Text))

Max Activating Token Index: 566

Text #19

Max Range: 0.6368. Min Range: -0.6368

Max Act: 0.4760. Min Act: -0.0001

Data Index: 690383 (C4 (Web Text))

Max Activating Token Index: 949