Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5694 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.8129. Min Act: -0.0000

Data Index: 10578 (C4 (Web Text))

Max Activating Token Index: 739

Text #1

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.7101. Min Act: -0.0000

Data Index: 1003671 (C4 (Web Text))

Max Activating Token Index: 107

Text #2

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.6881. Min Act: -0.0000

Data Index: 1169421 (C4 (Web Text))

Max Activating Token Index: 385

Text #3

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.5953. Min Act: -0.0000

Data Index: 179813 (C4 (Web Text))

Max Activating Token Index: 623

Text #4

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4512. Min Act: -0.0000

Data Index: 20648 (C4 (Web Text))

Max Activating Token Index: 488

Text #5

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4459. Min Act: -0.0000

Data Index: 1028519 (C4 (Web Text))

Max Activating Token Index: 622

Text #6

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4418. Min Act: -0.0000

Data Index: 943886 (C4 (Web Text))

Max Activating Token Index: 135

Text #7

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4257. Min Act: -0.0000

Data Index: 256956 (C4 (Web Text))

Max Activating Token Index: 672

Text #8

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.3966. Min Act: -0.0000

Data Index: 303211 (C4 (Web Text))

Max Activating Token Index: 542

Text #9

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4051. Min Act: -0.0000

Data Index: 677867 (C4 (Web Text))

Max Activating Token Index: 791

Text #10

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.3945. Min Act: -0.0000

Data Index: 1241932 (C4 (Web Text))

Max Activating Token Index: 963

Text #11

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.2160. Min Act: -0.0000

Data Index: 211904 (C4 (Web Text))

Max Activating Token Index: 847

Text #12

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4006. Min Act: -0.0000

Data Index: 259191 (C4 (Web Text))

Max Activating Token Index: 382

Text #13

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.2278. Min Act: -0.0000

Data Index: 320197 (C4 (Web Text))

Max Activating Token Index: 836

Text #14

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.3363. Min Act: -0.0000

Data Index: 978203 (C4 (Web Text))

Max Activating Token Index: 461

Text #15

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.4201. Min Act: -0.0000

Data Index: 1076628 (C4 (Web Text))

Max Activating Token Index: 521

Text #16

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.3144. Min Act: -0.0000

Data Index: 984365 (C4 (Web Text))

Max Activating Token Index: 845

Text #17

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.2780. Min Act: -0.0000

Data Index: 986092 (C4 (Web Text))

Max Activating Token Index: 858

Text #18

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.3594. Min Act: -0.0000

Data Index: 1221092 (C4 (Web Text))

Max Activating Token Index: 483

Text #19

Max Range: 1.8129. Min Range: -1.8129

Max Act: 1.2723. Min Act: -0.0000

Data Index: 1356726 (C4 (Web Text))

Max Activating Token Index: 843