Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 160 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0519. Min Act: -0.0000

Data Index: 196320 (C4 (Web Text))

Max Activating Token Index: 403

Text #1

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0524. Min Act: -0.0000

Data Index: 865609 (C4 (Web Text))

Max Activating Token Index: 608

Text #2

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0525. Min Act: -0.0001

Data Index: 972947 (C4 (Web Text))

Max Activating Token Index: 644

Text #3

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0527. Min Act: -0.0001

Data Index: 1249861 (C4 (Web Text))

Max Activating Token Index: 594

Text #4

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0482. Min Act: -0.0001

Data Index: 321269 (C4 (Web Text))

Max Activating Token Index: 432

Text #5

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0499. Min Act: -0.0001

Data Index: 237391 (C4 (Web Text))

Max Activating Token Index: 425

Text #6

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0506. Min Act: -0.0001

Data Index: 301244 (C4 (Web Text))

Max Activating Token Index: 796

Text #7

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0470. Min Act: -0.0001

Data Index: 1160823 (C4 (Web Text))

Max Activating Token Index: 48

Text #8

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0464. Min Act: -0.0001

Data Index: 566203 (C4 (Web Text))

Max Activating Token Index: 730

Text #9

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0478. Min Act: -0.0001

Data Index: 448185 (C4 (Web Text))

Max Activating Token Index: 598

Text #10

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0456. Min Act: -0.0001

Data Index: 410488 (C4 (Web Text))

Max Activating Token Index: 104

Text #11

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0463. Min Act: -0.0001

Data Index: 124223 (C4 (Web Text))

Max Activating Token Index: 915

Text #12

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0456. Min Act: -0.0001

Data Index: 1195715 (C4 (Web Text))

Max Activating Token Index: 264

Text #13

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0401. Min Act: -0.0001

Data Index: 1036177 (C4 (Web Text))

Max Activating Token Index: 237

Text #14

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0408. Min Act: -0.0001

Data Index: 333332 (C4 (Web Text))

Max Activating Token Index: 467

Text #15

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0364. Min Act: -0.0001

Data Index: 1084091 (C4 (Web Text))

Max Activating Token Index: 600

Text #16

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0419. Min Act: -0.0001

Data Index: 858248 (C4 (Web Text))

Max Activating Token Index: 489

Text #17

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0438. Min Act: -0.0001

Data Index: 805616 (C4 (Web Text))

Max Activating Token Index: 95

Text #18

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0402. Min Act: -0.0001

Data Index: 617735 (C4 (Web Text))

Max Activating Token Index: 255

Text #19

Max Range: 0.0527. Min Range: -0.0527

Max Act: 0.0409. Min Act: -0.0001

Data Index: 531886 (C4 (Web Text))

Max Activating Token Index: 627