Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4196 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.7730. Min Act: -0.0001

Data Index: 411986 (C4 (Web Text))

Max Activating Token Index: 862

Text #1

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.5720. Min Act: -0.0000

Data Index: 1135248 (C4 (Web Text))

Max Activating Token Index: 953

Text #2

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.4106. Min Act: -0.0000

Data Index: 921413 (C4 (Web Text))

Max Activating Token Index: 829

Text #3

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.4796. Min Act: -0.0000

Data Index: 4127 (C4 (Web Text))

Max Activating Token Index: 494

Text #4

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.3263. Min Act: -0.0000

Data Index: 1092156 (C4 (Web Text))

Max Activating Token Index: 270

Text #5

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.3479. Min Act: -0.0000

Data Index: 487834 (C4 (Web Text))

Max Activating Token Index: 52

Text #6

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.4585. Min Act: -0.0000

Data Index: 309283 (C4 (Web Text))

Max Activating Token Index: 446

Text #7

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.4873. Min Act: -0.0000

Data Index: 975467 (C4 (Web Text))

Max Activating Token Index: 1009

Text #8

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.3005. Min Act: -0.0001

Data Index: 1066401 (C4 (Web Text))

Max Activating Token Index: 690

Text #9

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.2301. Min Act: -0.0000

Data Index: 884763 (C4 (Web Text))

Max Activating Token Index: 882

Text #10

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.2567. Min Act: -0.0000

Data Index: 423953 (C4 (Web Text))

Max Activating Token Index: 50

Text #11

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.2224. Min Act: -0.0000

Data Index: 1157016 (C4 (Web Text))

Max Activating Token Index: 1006

Text #12

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.2184. Min Act: -0.0000

Data Index: 328407 (C4 (Web Text))

Max Activating Token Index: 907

Text #13

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.2140. Min Act: -0.0001

Data Index: 14533 (C4 (Web Text))

Max Activating Token Index: 411

Text #14

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.3378. Min Act: -0.0000

Data Index: 242816 (C4 (Web Text))

Max Activating Token Index: 919

Text #15

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.1106. Min Act: -0.0000

Data Index: 313319 (C4 (Web Text))

Max Activating Token Index: 892

Text #16

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.1528. Min Act: -0.0000

Data Index: 46548 (C4 (Web Text))

Max Activating Token Index: 169

Text #17

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.1863. Min Act: -0.0000

Data Index: 552932 (C4 (Web Text))

Max Activating Token Index: 209

Text #18

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.1101. Min Act: -0.0000

Data Index: 715398 (C4 (Web Text))

Max Activating Token Index: 375

Text #19

Max Range: 1.7730. Min Range: -1.7730

Max Act: 1.2255. Min Act: -0.0001

Data Index: 352007 (C4 (Web Text))

Max Activating Token Index: 277