Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4755 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.8532. Min Act: -0.0000

Data Index: 344506 (C4 (Web Text))

Max Activating Token Index: 540

Text #1

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6458. Min Act: -0.0000

Data Index: 366734 (C4 (Web Text))

Max Activating Token Index: 796

Text #2

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.7222. Min Act: -0.0000

Data Index: 734584 (C4 (Web Text))

Max Activating Token Index: 547

Text #3

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6956. Min Act: -0.0000

Data Index: 807909 (C4 (Web Text))

Max Activating Token Index: 890

Text #4

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6884. Min Act: -0.0000

Data Index: 1009324 (C4 (Web Text))

Max Activating Token Index: 358

Text #5

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6775. Min Act: -0.0000

Data Index: 1190708 (C4 (Web Text))

Max Activating Token Index: 163

Text #6

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6582. Min Act: -0.0000

Data Index: 1154324 (C4 (Web Text))

Max Activating Token Index: 578

Text #7

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6706. Min Act: -0.0000

Data Index: 46541 (C4 (Web Text))

Max Activating Token Index: 400

Text #8

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5831. Min Act: -0.0000

Data Index: 850003 (C4 (Web Text))

Max Activating Token Index: 273

Text #9

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6443. Min Act: -0.0000

Data Index: 629613 (C4 (Web Text))

Max Activating Token Index: 721

Text #10

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6280. Min Act: -0.0000

Data Index: 822190 (C4 (Web Text))

Max Activating Token Index: 208

Text #11

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5739. Min Act: -0.0000

Data Index: 270096 (C4 (Web Text))

Max Activating Token Index: 461

Text #12

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5913. Min Act: -0.0000

Data Index: 628140 (C4 (Web Text))

Max Activating Token Index: 638

Text #13

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5938. Min Act: -0.0000

Data Index: 186763 (C4 (Web Text))

Max Activating Token Index: 304

Text #14

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5368. Min Act: -0.0000

Data Index: 135840 (C4 (Web Text))

Max Activating Token Index: 447

Text #15

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5865. Min Act: -0.0000

Data Index: 472074 (C4 (Web Text))

Max Activating Token Index: 705

Text #16

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.4787. Min Act: -0.0000

Data Index: 310894 (C4 (Web Text))

Max Activating Token Index: 762

Text #17

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.5451. Min Act: -0.0000

Data Index: 572018 (C4 (Web Text))

Max Activating Token Index: 677

Text #18

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.4483. Min Act: -0.0000

Data Index: 862609 (C4 (Web Text))

Max Activating Token Index: 1022

Text #19

Max Range: 0.8532. Min Range: -0.8532

Max Act: 0.6080. Min Act: -0.0000

Data Index: 996514 (C4 (Web Text))

Max Activating Token Index: 785