Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1879 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.8491. Min Act: -0.0003

Data Index: 452996 (C4 (Web Text))

Max Activating Token Index: 28

Text #1

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.7658. Min Act: -0.0003

Data Index: 702537 (C4 (Web Text))

Max Activating Token Index: 488

Text #2

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.7129. Min Act: -0.0003

Data Index: 565882 (C4 (Web Text))

Max Activating Token Index: 326

Text #3

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.6507. Min Act: -0.0003

Data Index: 935778 (C4 (Web Text))

Max Activating Token Index: 748

Text #4

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.5806. Min Act: -0.0003

Data Index: 1086405 (C4 (Web Text))

Max Activating Token Index: 487

Text #5

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4661. Min Act: -0.0003

Data Index: 1148575 (C4 (Web Text))

Max Activating Token Index: 540

Text #6

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.5422. Min Act: -0.0003

Data Index: 772413 (C4 (Web Text))

Max Activating Token Index: 719

Text #7

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.5320. Min Act: -0.0003

Data Index: 1319825 (C4 (Web Text))

Max Activating Token Index: 48

Text #8

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4289. Min Act: -0.0003

Data Index: 1214449 (C4 (Web Text))

Max Activating Token Index: 78

Text #9

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4726. Min Act: -0.0003

Data Index: 936272 (C4 (Web Text))

Max Activating Token Index: 871

Text #10

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.5473. Min Act: -0.0003

Data Index: 1300142 (C4 (Web Text))

Max Activating Token Index: 491

Text #11

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4402. Min Act: -0.0003

Data Index: 931375 (C4 (Web Text))

Max Activating Token Index: 670

Text #12

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4387. Min Act: -0.0003

Data Index: 384221 (C4 (Web Text))

Max Activating Token Index: 482

Text #13

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4680. Min Act: -0.0003

Data Index: 1008080 (C4 (Web Text))

Max Activating Token Index: 732

Text #14

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4644. Min Act: -0.0003

Data Index: 38414 (C4 (Web Text))

Max Activating Token Index: 70

Text #15

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4292. Min Act: -0.0003

Data Index: 603147 (C4 (Web Text))

Max Activating Token Index: 711

Text #16

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4603. Min Act: -0.0003

Data Index: 1190843 (C4 (Web Text))

Max Activating Token Index: 1021

Text #17

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.3215. Min Act: -0.0003

Data Index: 31356 (C4 (Web Text))

Max Activating Token Index: 586

Text #18

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.4302. Min Act: -0.0003

Data Index: 282560 (C4 (Web Text))

Max Activating Token Index: 857

Text #19

Max Range: 1.8491. Min Range: -1.8491

Max Act: 1.3823. Min Act: -0.0003

Data Index: 99049 (C4 (Web Text))

Max Activating Token Index: 282