Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4354 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.7915. Min Act: -0.0000

Data Index: 982495 (C4 (Web Text))

Max Activating Token Index: 99

Text #1

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.3815. Min Act: -0.0000

Data Index: 393315 (C4 (Web Text))

Max Activating Token Index: 88

Text #2

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.3707. Min Act: -0.0000

Data Index: 182193 (C4 (Web Text))

Max Activating Token Index: 493

Text #3

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.4660. Min Act: -0.0000

Data Index: 1039687 (C4 (Web Text))

Max Activating Token Index: 564

Text #4

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.1448. Min Act: -0.0000

Data Index: 1097752 (C4 (Web Text))

Max Activating Token Index: 651

Text #5

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.2211. Min Act: -0.0000

Data Index: 904704 (C4 (Web Text))

Max Activating Token Index: 232

Text #6

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.3560. Min Act: -0.0000

Data Index: 986541 (C4 (Web Text))

Max Activating Token Index: 700

Text #7

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.1660. Min Act: -0.0000

Data Index: 50506 (C4 (Web Text))

Max Activating Token Index: 703

Text #8

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.0654. Min Act: -0.0000

Data Index: 25755 (C4 (Web Text))

Max Activating Token Index: 811

Text #9

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.0248. Min Act: -0.0000

Data Index: 185301 (C4 (Web Text))

Max Activating Token Index: 65

Text #10

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.9401. Min Act: -0.0000

Data Index: 257586 (C4 (Web Text))

Max Activating Token Index: 634

Text #11

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.8822. Min Act: -0.0000

Data Index: 83150 (C4 (Web Text))

Max Activating Token Index: 564

Text #12

Max Range: 2.7915. Min Range: -2.7915

Max Act: 2.0010. Min Act: -0.0000

Data Index: 1005333 (C4 (Web Text))

Max Activating Token Index: 642

Text #13

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.8938. Min Act: -0.0000

Data Index: 422044 (C4 (Web Text))

Max Activating Token Index: 220

Text #14

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.9568. Min Act: -0.0000

Data Index: 617397 (C4 (Web Text))

Max Activating Token Index: 968

Text #15

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.9272. Min Act: -0.0000

Data Index: 1125317 (C4 (Web Text))

Max Activating Token Index: 117

Text #16

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.8037. Min Act: -0.0000

Data Index: 350166 (C4 (Web Text))

Max Activating Token Index: 475

Text #17

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.9049. Min Act: -0.0000

Data Index: 955609 (C4 (Web Text))

Max Activating Token Index: 972

Text #18

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.8915. Min Act: -0.0000

Data Index: 1084586 (C4 (Web Text))

Max Activating Token Index: 205

Text #19

Max Range: 2.7915. Min Range: -2.7915

Max Act: 1.8758. Min Act: -0.0000

Data Index: 799217 (C4 (Web Text))

Max Activating Token Index: 776