Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2124 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1184. Min Range: -2.1184

Max Act: 2.1184. Min Act: -0.0000

Data Index: 753996 (C4 (Web Text))

Max Activating Token Index: 186

Text #1

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.7443. Min Act: -0.0000

Data Index: 310683 (C4 (Web Text))

Max Activating Token Index: 846

Text #2

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.7557. Min Act: -0.0000

Data Index: 286351 (C4 (Web Text))

Max Activating Token Index: 888

Text #3

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.6608. Min Act: -0.0000

Data Index: 1277654 (C4 (Web Text))

Max Activating Token Index: 574

Text #4

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5934. Min Act: -0.0000

Data Index: 1558455 (Python Code)

Max Activating Token Index: 243

Text #5

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.7756. Min Act: -0.0000

Data Index: 656153 (C4 (Web Text))

Max Activating Token Index: 688

Text #6

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.6550. Min Act: -0.0000

Data Index: 824842 (C4 (Web Text))

Max Activating Token Index: 440

Text #7

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5203. Min Act: -0.0000

Data Index: 1319837 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.6153. Min Act: -0.0000

Data Index: 1337172 (C4 (Web Text))

Max Activating Token Index: 907

Text #9

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5781. Min Act: -0.0000

Data Index: 866953 (C4 (Web Text))

Max Activating Token Index: 855

Text #10

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.4070. Min Act: -0.0000

Data Index: 716531 (C4 (Web Text))

Max Activating Token Index: 279

Text #11

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.4865. Min Act: -0.0000

Data Index: 83861 (C4 (Web Text))

Max Activating Token Index: 603

Text #12

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5498. Min Act: -0.0000

Data Index: 733091 (C4 (Web Text))

Max Activating Token Index: 863

Text #13

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.4446. Min Act: -0.0000

Data Index: 1232272 (C4 (Web Text))

Max Activating Token Index: 924

Text #14

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5389. Min Act: -0.0000

Data Index: 1032504 (C4 (Web Text))

Max Activating Token Index: 48

Text #15

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5872. Min Act: -0.0000

Data Index: 158677 (C4 (Web Text))

Max Activating Token Index: 375

Text #16

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.6037. Min Act: -0.0000

Data Index: 1459170 (Python Code)

Max Activating Token Index: 731

Text #17

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5094. Min Act: -0.0000

Data Index: 112892 (C4 (Web Text))

Max Activating Token Index: 287

Text #18

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.5536. Min Act: -0.0000

Data Index: 1028500 (C4 (Web Text))

Max Activating Token Index: 843

Text #19

Max Range: 2.1184. Min Range: -2.1184

Max Act: 1.4601. Min Act: -0.0000

Data Index: 1554632 (Python Code)

Max Activating Token Index: 159