Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3435 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.5374. Min Act: -0.0000

Data Index: 651465 (C4 (Web Text))

Max Activating Token Index: 832

Text #1

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4972. Min Act: -0.0001

Data Index: 1065188 (C4 (Web Text))

Max Activating Token Index: 792

Text #2

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4549. Min Act: -0.0001

Data Index: 1340506 (C4 (Web Text))

Max Activating Token Index: 642

Text #3

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4463. Min Act: -0.0000

Data Index: 414934 (C4 (Web Text))

Max Activating Token Index: 372

Text #4

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4145. Min Act: -0.0000

Data Index: 167830 (C4 (Web Text))

Max Activating Token Index: 754

Text #5

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4412. Min Act: -0.0001

Data Index: 648049 (C4 (Web Text))

Max Activating Token Index: 954

Text #6

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4283. Min Act: -0.0000

Data Index: 944076 (C4 (Web Text))

Max Activating Token Index: 582

Text #7

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4579. Min Act: -0.0000

Data Index: 572494 (C4 (Web Text))

Max Activating Token Index: 210

Text #8

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4572. Min Act: -0.0000

Data Index: 393129 (C4 (Web Text))

Max Activating Token Index: 960

Text #9

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4197. Min Act: -0.0000

Data Index: 1107940 (C4 (Web Text))

Max Activating Token Index: 568

Text #10

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4198. Min Act: -0.0000

Data Index: 523499 (C4 (Web Text))

Max Activating Token Index: 433

Text #11

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4269. Min Act: -0.0000

Data Index: 843564 (C4 (Web Text))

Max Activating Token Index: 986

Text #12

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4408. Min Act: -0.0001

Data Index: 1317522 (C4 (Web Text))

Max Activating Token Index: 349

Text #13

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4267. Min Act: -0.0000

Data Index: 944043 (C4 (Web Text))

Max Activating Token Index: 91

Text #14

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4221. Min Act: -0.0000

Data Index: 413619 (C4 (Web Text))

Max Activating Token Index: 572

Text #15

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4055. Min Act: -0.0001

Data Index: 123065 (C4 (Web Text))

Max Activating Token Index: 226

Text #16

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4185. Min Act: -0.0001

Data Index: 847740 (C4 (Web Text))

Max Activating Token Index: 524

Text #17

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4183. Min Act: -0.0000

Data Index: 533415 (C4 (Web Text))

Max Activating Token Index: 782

Text #18

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4168. Min Act: -0.0000

Data Index: 1034216 (C4 (Web Text))

Max Activating Token Index: 192

Text #19

Max Range: 0.5374. Min Range: -0.5374

Max Act: 0.4166. Min Act: -0.0001

Data Index: 329063 (C4 (Web Text))

Max Activating Token Index: 539