Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2236 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.5200. Min Act: -0.0000

Data Index: 1055524 (C4 (Web Text))

Max Activating Token Index: 742

Text #1

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.5419. Min Act: -0.0000

Data Index: 989960 (C4 (Web Text))

Max Activating Token Index: 555

Text #2

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.5556. Min Act: -0.0000

Data Index: 60779 (C4 (Web Text))

Max Activating Token Index: 727

Text #3

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.5258. Min Act: -0.0000

Data Index: 307641 (C4 (Web Text))

Max Activating Token Index: 267

Text #4

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.5115. Min Act: -0.0000

Data Index: 280821 (C4 (Web Text))

Max Activating Token Index: 99

Text #5

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4816. Min Act: -0.0000

Data Index: 388869 (C4 (Web Text))

Max Activating Token Index: 350

Text #6

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4994. Min Act: -0.0000

Data Index: 1158733 (C4 (Web Text))

Max Activating Token Index: 760

Text #7

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.5175. Min Act: -0.0000

Data Index: 1103004 (C4 (Web Text))

Max Activating Token Index: 715

Text #8

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4520. Min Act: -0.0000

Data Index: 692356 (C4 (Web Text))

Max Activating Token Index: 230

Text #9

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4835. Min Act: -0.0000

Data Index: 1224077 (C4 (Web Text))

Max Activating Token Index: 480

Text #10

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4537. Min Act: -0.0000

Data Index: 1044906 (C4 (Web Text))

Max Activating Token Index: 747

Text #11

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4453. Min Act: -0.0000

Data Index: 67425 (C4 (Web Text))

Max Activating Token Index: 323

Text #12

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4328. Min Act: -0.0000

Data Index: 497599 (C4 (Web Text))

Max Activating Token Index: 715

Text #13

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4417. Min Act: -0.0000

Data Index: 24263 (C4 (Web Text))

Max Activating Token Index: 433

Text #14

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4550. Min Act: -0.0000

Data Index: 1261008 (C4 (Web Text))

Max Activating Token Index: 730

Text #15

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4384. Min Act: -0.0000

Data Index: 135756 (C4 (Web Text))

Max Activating Token Index: 393

Text #16

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4120. Min Act: -0.0000

Data Index: 1286625 (C4 (Web Text))

Max Activating Token Index: 337

Text #17

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4594. Min Act: -0.0000

Data Index: 540503 (C4 (Web Text))

Max Activating Token Index: 231

Text #18

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4194. Min Act: -0.0000

Data Index: 978960 (C4 (Web Text))

Max Activating Token Index: 389

Text #19

Max Range: 0.5556. Min Range: -0.5556

Max Act: 0.4351. Min Act: -0.0000

Data Index: 254690 (C4 (Web Text))

Max Activating Token Index: 590