Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4300 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.4485. Min Range: -3.4485

Max Act: 3.4485. Min Act: -0.0000

Data Index: 1478171 (Python Code)

Max Activating Token Index: 651

Text #1

Max Range: 3.4485. Min Range: -3.4485

Max Act: 3.2631. Min Act: -0.0000

Data Index: 1654196 (Python Code)

Max Activating Token Index: 519

Text #2

Max Range: 3.4485. Min Range: -3.4485

Max Act: 3.1593. Min Act: -0.0000

Data Index: 1294501 (C4 (Web Text))

Max Activating Token Index: 339

Text #3

Max Range: 3.4485. Min Range: -3.4485

Max Act: 3.0624. Min Act: -0.0000

Data Index: 40960 (C4 (Web Text))

Max Activating Token Index: 369

Text #4

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.6301. Min Act: -0.0000

Data Index: 196922 (C4 (Web Text))

Max Activating Token Index: 603

Text #5

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.8867. Min Act: -0.0000

Data Index: 827973 (C4 (Web Text))

Max Activating Token Index: 219

Text #6

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.8806. Min Act: -0.0000

Data Index: 1556614 (Python Code)

Max Activating Token Index: 86

Text #7

Max Range: 3.4485. Min Range: -3.4485

Max Act: 3.0228. Min Act: -0.0000

Data Index: 1188623 (C4 (Web Text))

Max Activating Token Index: 971

Text #8

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7463. Min Act: -0.0000

Data Index: 968656 (C4 (Web Text))

Max Activating Token Index: 696

Text #9

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7417. Min Act: -0.0000

Data Index: 723411 (C4 (Web Text))

Max Activating Token Index: 683

Text #10

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.6877. Min Act: -0.0000

Data Index: 1213908 (C4 (Web Text))

Max Activating Token Index: 993

Text #11

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.8611. Min Act: -0.0000

Data Index: 1159569 (C4 (Web Text))

Max Activating Token Index: 201

Text #12

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7046. Min Act: -0.0000

Data Index: 419475 (C4 (Web Text))

Max Activating Token Index: 760

Text #13

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7921. Min Act: -0.0000

Data Index: 1035894 (C4 (Web Text))

Max Activating Token Index: 836

Text #14

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7834. Min Act: -0.0000

Data Index: 163249 (C4 (Web Text))

Max Activating Token Index: 424

Text #15

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.9201. Min Act: -0.0000

Data Index: 168274 (C4 (Web Text))

Max Activating Token Index: 134

Text #16

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.6302. Min Act: -0.0000

Data Index: 108165 (C4 (Web Text))

Max Activating Token Index: 536

Text #17

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7543. Min Act: -0.0000

Data Index: 648286 (C4 (Web Text))

Max Activating Token Index: 991

Text #18

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.4202. Min Act: -0.0000

Data Index: 641366 (C4 (Web Text))

Max Activating Token Index: 734

Text #19

Max Range: 3.4485. Min Range: -3.4485

Max Act: 2.7201. Min Act: -0.0000

Data Index: 153213 (C4 (Web Text))

Max Activating Token Index: 241