Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 961 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.6519. Min Act: -0.0002

Data Index: 691103 (C4 (Web Text))

Max Activating Token Index: 849

Text #1

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4280. Min Act: -0.0002

Data Index: 238533 (C4 (Web Text))

Max Activating Token Index: 761

Text #2

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.3264. Min Act: -0.0002

Data Index: 529489 (C4 (Web Text))

Max Activating Token Index: 386

Text #3

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.6179. Min Act: -0.0002

Data Index: 862996 (C4 (Web Text))

Max Activating Token Index: 790

Text #4

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4932. Min Act: -0.0002

Data Index: 395353 (C4 (Web Text))

Max Activating Token Index: 152

Text #5

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4245. Min Act: -0.0002

Data Index: 680767 (C4 (Web Text))

Max Activating Token Index: 65

Text #6

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4058. Min Act: -0.0002

Data Index: 1122614 (C4 (Web Text))

Max Activating Token Index: 626

Text #7

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.7010. Min Act: -0.0002

Data Index: 105159 (C4 (Web Text))

Max Activating Token Index: 102

Text #8

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4735. Min Act: -0.0002

Data Index: 1015369 (C4 (Web Text))

Max Activating Token Index: 80

Text #9

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.3784. Min Act: -0.0002

Data Index: 1220350 (C4 (Web Text))

Max Activating Token Index: 571

Text #10

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.6278. Min Act: -0.0002

Data Index: 155435 (C4 (Web Text))

Max Activating Token Index: 315

Text #11

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4532. Min Act: -0.0002

Data Index: 388939 (C4 (Web Text))

Max Activating Token Index: 202

Text #12

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.5889. Min Act: -0.0002

Data Index: 10199 (C4 (Web Text))

Max Activating Token Index: 125

Text #13

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4575. Min Act: -0.0002

Data Index: 993847 (C4 (Web Text))

Max Activating Token Index: 4

Text #14

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.4168. Min Act: -0.0002

Data Index: 681793 (C4 (Web Text))

Max Activating Token Index: 923

Text #15

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.5991. Min Act: -0.0002

Data Index: 669180 (C4 (Web Text))

Max Activating Token Index: 701

Text #16

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.5254. Min Act: -0.0002

Data Index: 52360 (C4 (Web Text))

Max Activating Token Index: 50

Text #17

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.5544. Min Act: -0.0002

Data Index: 824959 (C4 (Web Text))

Max Activating Token Index: 824

Text #18

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.5049. Min Act: -0.0002

Data Index: 1219855 (C4 (Web Text))

Max Activating Token Index: 716

Text #19

Max Range: 1.7010. Min Range: -1.7010

Max Act: 1.5137. Min Act: -0.0002

Data Index: 442135 (C4 (Web Text))

Max Activating Token Index: 643