Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3805 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1078. Min Range: -1.1078

Max Act: 1.0846. Min Act: -0.0000

Data Index: 616526 (C4 (Web Text))

Max Activating Token Index: 754

Text #1

Max Range: 1.1078. Min Range: -1.1078

Max Act: 1.1078. Min Act: -0.0000

Data Index: 1219754 (C4 (Web Text))

Max Activating Token Index: 552

Text #2

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.9724. Min Act: -0.0000

Data Index: 261334 (C4 (Web Text))

Max Activating Token Index: 480

Text #3

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.8689. Min Act: -0.0000

Data Index: 1095840 (C4 (Web Text))

Max Activating Token Index: 712

Text #4

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.9526. Min Act: -0.0000

Data Index: 549020 (C4 (Web Text))

Max Activating Token Index: 965

Text #5

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.9328. Min Act: -0.0000

Data Index: 217410 (C4 (Web Text))

Max Activating Token Index: 967

Text #6

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.8288. Min Act: -0.0000

Data Index: 12409 (C4 (Web Text))

Max Activating Token Index: 445

Text #7

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.7166. Min Act: -0.0000

Data Index: 492562 (C4 (Web Text))

Max Activating Token Index: 932

Text #8

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.7669. Min Act: -0.0000

Data Index: 404329 (C4 (Web Text))

Max Activating Token Index: 159

Text #9

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.7089. Min Act: -0.0000

Data Index: 476311 (C4 (Web Text))

Max Activating Token Index: 1018

Text #10

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6061. Min Act: -0.0000

Data Index: 1137091 (C4 (Web Text))

Max Activating Token Index: 654

Text #11

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6239. Min Act: -0.0000

Data Index: 303403 (C4 (Web Text))

Max Activating Token Index: 507

Text #12

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6432. Min Act: -0.0000

Data Index: 135914 (C4 (Web Text))

Max Activating Token Index: 122

Text #13

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6141. Min Act: -0.0000

Data Index: 648119 (C4 (Web Text))

Max Activating Token Index: 655

Text #14

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6246. Min Act: -0.0000

Data Index: 579320 (C4 (Web Text))

Max Activating Token Index: 241

Text #15

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6308. Min Act: -0.0000

Data Index: 105326 (C4 (Web Text))

Max Activating Token Index: 776

Text #16

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.7357. Min Act: -0.0000

Data Index: 1058856 (C4 (Web Text))

Max Activating Token Index: 897

Text #17

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.6009. Min Act: -0.0000

Data Index: 258343 (C4 (Web Text))

Max Activating Token Index: 865

Text #18

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.5798. Min Act: -0.0000

Data Index: 409830 (C4 (Web Text))

Max Activating Token Index: 82

Text #19

Max Range: 1.1078. Min Range: -1.1078

Max Act: 0.5830. Min Act: -0.0000

Data Index: 697562 (C4 (Web Text))

Max Activating Token Index: 130