Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1818 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.3119. Min Range: -3.3119

Max Act: 3.3119. Min Act: -0.0000

Data Index: 513193 (C4 (Web Text))

Max Activating Token Index: 181

Text #1

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.8004. Min Act: -0.0000

Data Index: 997511 (C4 (Web Text))

Max Activating Token Index: 888

Text #2

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.6329. Min Act: -0.0000

Data Index: 386919 (C4 (Web Text))

Max Activating Token Index: 618

Text #3

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.6193. Min Act: -0.0000

Data Index: 28116 (C4 (Web Text))

Max Activating Token Index: 245

Text #4

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.5423. Min Act: -0.0000

Data Index: 454803 (C4 (Web Text))

Max Activating Token Index: 1012

Text #5

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.5920. Min Act: -0.0000

Data Index: 816185 (C4 (Web Text))

Max Activating Token Index: 553

Text #6

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.4461. Min Act: -0.0000

Data Index: 125909 (C4 (Web Text))

Max Activating Token Index: 586

Text #7

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.3452. Min Act: -0.0000

Data Index: 907057 (C4 (Web Text))

Max Activating Token Index: 440

Text #8

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.4081. Min Act: -0.0000

Data Index: 885476 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.4386. Min Act: -0.0000

Data Index: 1180650 (C4 (Web Text))

Max Activating Token Index: 699

Text #10

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.1291. Min Act: -0.0000

Data Index: 258593 (C4 (Web Text))

Max Activating Token Index: 856

Text #11

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.1150. Min Act: -0.0000

Data Index: 1052729 (C4 (Web Text))

Max Activating Token Index: 523

Text #12

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.0006. Min Act: -0.0000

Data Index: 1018197 (C4 (Web Text))

Max Activating Token Index: 426

Text #13

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.2237. Min Act: -0.0000

Data Index: 744523 (C4 (Web Text))

Max Activating Token Index: 30

Text #14

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.1914. Min Act: -0.0000

Data Index: 84308 (C4 (Web Text))

Max Activating Token Index: 707

Text #15

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.1036. Min Act: -0.0000

Data Index: 1033122 (C4 (Web Text))

Max Activating Token Index: 73

Text #16

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.3086. Min Act: -0.0000

Data Index: 1320686 (C4 (Web Text))

Max Activating Token Index: 821

Text #17

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.0552. Min Act: -0.0000

Data Index: 852748 (C4 (Web Text))

Max Activating Token Index: 74

Text #18

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.0493. Min Act: -0.0000

Data Index: 1168597 (C4 (Web Text))

Max Activating Token Index: 241

Text #19

Max Range: 3.3119. Min Range: -3.3119

Max Act: 2.1227. Min Act: -0.0000

Data Index: 1056608 (C4 (Web Text))

Max Activating Token Index: 871