Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4958 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2528. Min Range: -1.2528

Max Act: 1.2528. Min Act: -0.0000

Data Index: 827216 (C4 (Web Text))

Max Activating Token Index: 600

Text #1

Max Range: 1.2528. Min Range: -1.2528

Max Act: 1.1125. Min Act: -0.0000

Data Index: 1356324 (C4 (Web Text))

Max Activating Token Index: 268

Text #2

Max Range: 1.2528. Min Range: -1.2528

Max Act: 1.2095. Min Act: -0.0000

Data Index: 421207 (C4 (Web Text))

Max Activating Token Index: 507

Text #3

Max Range: 1.2528. Min Range: -1.2528

Max Act: 1.1599. Min Act: -0.0000

Data Index: 366541 (C4 (Web Text))

Max Activating Token Index: 616

Text #4

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.9635. Min Act: -0.0000

Data Index: 958434 (C4 (Web Text))

Max Activating Token Index: 109

Text #5

Max Range: 1.2528. Min Range: -1.2528

Max Act: 1.1553. Min Act: -0.0000

Data Index: 1105163 (C4 (Web Text))

Max Activating Token Index: 587

Text #6

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.9388. Min Act: -0.0000

Data Index: 958818 (C4 (Web Text))

Max Activating Token Index: 327

Text #7

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.9940. Min Act: -0.0000

Data Index: 1038251 (C4 (Web Text))

Max Activating Token Index: 259

Text #8

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.9770. Min Act: -0.0000

Data Index: 1118673 (C4 (Web Text))

Max Activating Token Index: 433

Text #9

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.7583. Min Act: -0.0000

Data Index: 683489 (C4 (Web Text))

Max Activating Token Index: 262

Text #10

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.8772. Min Act: -0.0000

Data Index: 1320029 (C4 (Web Text))

Max Activating Token Index: 977

Text #11

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.8204. Min Act: -0.0000

Data Index: 368630 (C4 (Web Text))

Max Activating Token Index: 270

Text #12

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.8022. Min Act: -0.0000

Data Index: 148592 (C4 (Web Text))

Max Activating Token Index: 518

Text #13

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.8762. Min Act: -0.0000

Data Index: 1189510 (C4 (Web Text))

Max Activating Token Index: 598

Text #14

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.8196. Min Act: -0.0000

Data Index: 369261 (C4 (Web Text))

Max Activating Token Index: 223

Text #15

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.7320. Min Act: -0.0000

Data Index: 1300749 (C4 (Web Text))

Max Activating Token Index: 230

Text #16

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.7556. Min Act: -0.0000

Data Index: 1331208 (C4 (Web Text))

Max Activating Token Index: 150

Text #17

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.7584. Min Act: -0.0000

Data Index: 825381 (C4 (Web Text))

Max Activating Token Index: 483

Text #18

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.7321. Min Act: -0.0000

Data Index: 608 (C4 (Web Text))

Max Activating Token Index: 454

Text #19

Max Range: 1.2528. Min Range: -1.2528

Max Act: 0.7717. Min Act: -0.0000

Data Index: 850720 (C4 (Web Text))

Max Activating Token Index: 596