Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 603 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 15.4182. Min Range: -15.4182

Max Act: 15.4182. Min Act: -0.0003

Data Index: 231081 (C4 (Web Text))

Max Activating Token Index: 372

Text #1

Max Range: 15.4182. Min Range: -15.4182

Max Act: 15.3566. Min Act: -0.0002

Data Index: 1267459 (C4 (Web Text))

Max Activating Token Index: 893

Text #2

Max Range: 15.4182. Min Range: -15.4182

Max Act: 15.1034. Min Act: -0.0003

Data Index: 1258856 (C4 (Web Text))

Max Activating Token Index: 550

Text #3

Max Range: 15.4182. Min Range: -15.4182

Max Act: 15.0995. Min Act: -0.0002

Data Index: 490576 (C4 (Web Text))

Max Activating Token Index: 694

Text #4

Max Range: 15.4182. Min Range: -15.4182

Max Act: 15.0393. Min Act: -0.0003

Data Index: 655244 (C4 (Web Text))

Max Activating Token Index: 803

Text #5

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9754. Min Act: -0.0002

Data Index: 893906 (C4 (Web Text))

Max Activating Token Index: 454

Text #6

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9127. Min Act: -0.0003

Data Index: 1075927 (C4 (Web Text))

Max Activating Token Index: 666

Text #7

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9117. Min Act: -0.0003

Data Index: 416098 (C4 (Web Text))

Max Activating Token Index: 616

Text #8

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9108. Min Act: -0.0003

Data Index: 906880 (C4 (Web Text))

Max Activating Token Index: 173

Text #9

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9718. Min Act: -0.0003

Data Index: 811055 (C4 (Web Text))

Max Activating Token Index: 443

Text #10

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9150. Min Act: -0.0003

Data Index: 48414 (C4 (Web Text))

Max Activating Token Index: 768

Text #11

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9126. Min Act: -0.0003

Data Index: 440212 (C4 (Web Text))

Max Activating Token Index: 614

Text #12

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9121. Min Act: -0.0003

Data Index: 1183087 (C4 (Web Text))

Max Activating Token Index: 592

Text #13

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9113. Min Act: -0.0002

Data Index: 301902 (C4 (Web Text))

Max Activating Token Index: 410

Text #14

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9107. Min Act: -0.0002

Data Index: 559017 (C4 (Web Text))

Max Activating Token Index: 779

Text #15

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9103. Min Act: -0.0003

Data Index: 993239 (C4 (Web Text))

Max Activating Token Index: 462

Text #16

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9074. Min Act: -0.0003

Data Index: 27781 (C4 (Web Text))

Max Activating Token Index: 578

Text #17

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.9072. Min Act: -0.0003

Data Index: 370685 (C4 (Web Text))

Max Activating Token Index: 509

Text #18

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.8330. Min Act: -0.0003

Data Index: 1533128 (Python Code)

Max Activating Token Index: 1

Text #19

Max Range: 15.4182. Min Range: -15.4182

Max Act: 14.8330. Min Act: -0.0002

Data Index: 1460277 (Python Code)

Max Activating Token Index: 1