Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2829 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.6029. Min Act: -0.0000

Data Index: 1618863 (Python Code)

Max Activating Token Index: 286

Text #1

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5887. Min Act: -0.0000

Data Index: 1068833 (C4 (Web Text))

Max Activating Token Index: 1018

Text #2

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5966. Min Act: -0.0000

Data Index: 1125533 (C4 (Web Text))

Max Activating Token Index: 875

Text #3

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5958. Min Act: -0.0000

Data Index: 93024 (C4 (Web Text))

Max Activating Token Index: 279

Text #4

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5625. Min Act: -0.0000

Data Index: 1185324 (C4 (Web Text))

Max Activating Token Index: 125

Text #5

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.6149. Min Act: -0.0000

Data Index: 1640927 (Python Code)

Max Activating Token Index: 904

Text #6

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5885. Min Act: -0.0000

Data Index: 1656921 (Python Code)

Max Activating Token Index: 492

Text #7

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5510. Min Act: -0.0000

Data Index: 6327 (C4 (Web Text))

Max Activating Token Index: 821

Text #8

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5627. Min Act: -0.0000

Data Index: 508 (C4 (Web Text))

Max Activating Token Index: 914

Text #9

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5608. Min Act: -0.0000

Data Index: 1226401 (C4 (Web Text))

Max Activating Token Index: 1003

Text #10

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5695. Min Act: -0.0000

Data Index: 1540772 (Python Code)

Max Activating Token Index: 338

Text #11

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5494. Min Act: -0.0000

Data Index: 1169605 (C4 (Web Text))

Max Activating Token Index: 538

Text #12

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5533. Min Act: -0.0000

Data Index: 555142 (C4 (Web Text))

Max Activating Token Index: 565

Text #13

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5890. Min Act: -0.0000

Data Index: 333038 (C4 (Web Text))

Max Activating Token Index: 309

Text #14

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5301. Min Act: -0.0000

Data Index: 1584769 (Python Code)

Max Activating Token Index: 122

Text #15

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5158. Min Act: -0.0000

Data Index: 933045 (C4 (Web Text))

Max Activating Token Index: 551

Text #16

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5694. Min Act: -0.0000

Data Index: 1502885 (Python Code)

Max Activating Token Index: 529

Text #17

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5296. Min Act: -0.0000

Data Index: 1179617 (C4 (Web Text))

Max Activating Token Index: 634

Text #18

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5237. Min Act: -0.0000

Data Index: 625072 (C4 (Web Text))

Max Activating Token Index: 1018

Text #19

Max Range: 0.6149. Min Range: -0.6149

Max Act: 0.5081. Min Act: -0.0000

Data Index: 1056840 (C4 (Web Text))

Max Activating Token Index: 287