Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3200 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.5197. Min Act: -0.0000

Data Index: 1165544 (C4 (Web Text))

Max Activating Token Index: 821

Text #1

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.3373. Min Act: -0.0000

Data Index: 1032854 (C4 (Web Text))

Max Activating Token Index: 891

Text #2

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.4125. Min Act: -0.0000

Data Index: 193706 (C4 (Web Text))

Max Activating Token Index: 273

Text #3

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.6157. Min Act: -0.0000

Data Index: 1121043 (C4 (Web Text))

Max Activating Token Index: 102

Text #4

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.2579. Min Act: -0.0000

Data Index: 1007256 (C4 (Web Text))

Max Activating Token Index: 1012

Text #5

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.9367. Min Act: -0.0000

Data Index: 152801 (C4 (Web Text))

Max Activating Token Index: 156

Text #6

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.0311. Min Act: -0.0000

Data Index: 184982 (C4 (Web Text))

Max Activating Token Index: 250

Text #7

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.1497. Min Act: -0.0000

Data Index: 1322997 (C4 (Web Text))

Max Activating Token Index: 893

Text #8

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.2338. Min Act: -0.0000

Data Index: 797959 (C4 (Web Text))

Max Activating Token Index: 404

Text #9

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.8510. Min Act: -0.0000

Data Index: 1115034 (C4 (Web Text))

Max Activating Token Index: 145

Text #10

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.8605. Min Act: -0.0000

Data Index: 543936 (C4 (Web Text))

Max Activating Token Index: 487

Text #11

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.8109. Min Act: -0.0000

Data Index: 429752 (C4 (Web Text))

Max Activating Token Index: 868

Text #12

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.8056. Min Act: -0.0000

Data Index: 1223502 (C4 (Web Text))

Max Activating Token Index: 991

Text #13

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.0998. Min Act: -0.0000

Data Index: 669945 (C4 (Web Text))

Max Activating Token Index: 808

Text #14

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.8647. Min Act: -0.0000

Data Index: 766248 (C4 (Web Text))

Max Activating Token Index: 399

Text #15

Max Range: 2.6157. Min Range: -2.6157

Max Act: 2.0678. Min Act: -0.0000

Data Index: 504964 (C4 (Web Text))

Max Activating Token Index: 911

Text #16

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.8551. Min Act: -0.0000

Data Index: 960497 (C4 (Web Text))

Max Activating Token Index: 1012

Text #17

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.5389. Min Act: -0.0000

Data Index: 1005917 (C4 (Web Text))

Max Activating Token Index: 623

Text #18

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.7251. Min Act: -0.0000

Data Index: 587440 (C4 (Web Text))

Max Activating Token Index: 893

Text #19

Max Range: 2.6157. Min Range: -2.6157

Max Act: 1.7384. Min Act: -0.0000

Data Index: 493148 (C4 (Web Text))

Max Activating Token Index: 856