Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3905 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9672. Min Act: -0.0000

Data Index: 1186789 (C4 (Web Text))

Max Activating Token Index: 724

Text #1

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9732. Min Act: -0.0000

Data Index: 169380 (C4 (Web Text))

Max Activating Token Index: 512

Text #2

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9609. Min Act: -0.0000

Data Index: 581582 (C4 (Web Text))

Max Activating Token Index: 361

Text #3

Max Range: 2.1278. Min Range: -2.1278

Max Act: 2.0608. Min Act: -0.0000

Data Index: 74128 (C4 (Web Text))

Max Activating Token Index: 453

Text #4

Max Range: 2.1278. Min Range: -2.1278

Max Act: 2.1278. Min Act: -0.0000

Data Index: 344395 (C4 (Web Text))

Max Activating Token Index: 96

Text #5

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.8698. Min Act: -0.0000

Data Index: 220255 (C4 (Web Text))

Max Activating Token Index: 185

Text #6

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9736. Min Act: -0.0000

Data Index: 944582 (C4 (Web Text))

Max Activating Token Index: 237

Text #7

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9552. Min Act: -0.0000

Data Index: 958690 (C4 (Web Text))

Max Activating Token Index: 338

Text #8

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.8689. Min Act: -0.0000

Data Index: 1239526 (C4 (Web Text))

Max Activating Token Index: 491

Text #9

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9611. Min Act: -0.0000

Data Index: 693366 (C4 (Web Text))

Max Activating Token Index: 365

Text #10

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9622. Min Act: -0.0000

Data Index: 738823 (C4 (Web Text))

Max Activating Token Index: 876

Text #11

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9507. Min Act: -0.0000

Data Index: 972016 (C4 (Web Text))

Max Activating Token Index: 885

Text #12

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9434. Min Act: -0.0000

Data Index: 441236 (C4 (Web Text))

Max Activating Token Index: 736

Text #13

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9605. Min Act: -0.0000

Data Index: 223358 (C4 (Web Text))

Max Activating Token Index: 236

Text #14

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9152. Min Act: -0.0000

Data Index: 746905 (C4 (Web Text))

Max Activating Token Index: 291

Text #15

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9031. Min Act: -0.0000

Data Index: 1337230 (C4 (Web Text))

Max Activating Token Index: 639

Text #16

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.7078. Min Act: -0.0000

Data Index: 869567 (C4 (Web Text))

Max Activating Token Index: 848

Text #17

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.9025. Min Act: -0.0000

Data Index: 468407 (C4 (Web Text))

Max Activating Token Index: 340

Text #18

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.7659. Min Act: -0.0000

Data Index: 1155118 (C4 (Web Text))

Max Activating Token Index: 1021

Text #19

Max Range: 2.1278. Min Range: -2.1278

Max Act: 1.8658. Min Act: -0.0000

Data Index: 417758 (C4 (Web Text))

Max Activating Token Index: 976