Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1334 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 3.6543. Min Range: -3.6543

Max Act: 3.6543. Min Act: -0.0001

Data Index: 808973 (C4 (Web Text))

Max Activating Token Index: 263

Text #1

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.7959. Min Act: -0.0001

Data Index: 821763 (C4 (Web Text))

Max Activating Token Index: 374

Text #2

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.8116. Min Act: -0.0001

Data Index: 164581 (C4 (Web Text))

Max Activating Token Index: 860

Text #3

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.4538. Min Act: -0.0001

Data Index: 1117349 (C4 (Web Text))

Max Activating Token Index: 241

Text #4

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.3546. Min Act: -0.0001

Data Index: 185776 (C4 (Web Text))

Max Activating Token Index: 491

Text #5

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.1018. Min Act: -0.0001

Data Index: 854282 (C4 (Web Text))

Max Activating Token Index: 500

Text #6

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.1563. Min Act: -0.0001

Data Index: 1341786 (C4 (Web Text))

Max Activating Token Index: 740

Text #7

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.0299. Min Act: -0.0001

Data Index: 779654 (C4 (Web Text))

Max Activating Token Index: 454

Text #8

Max Range: 3.6543. Min Range: -3.6543

Max Act: 2.1363. Min Act: -0.0001

Data Index: 1003135 (C4 (Web Text))

Max Activating Token Index: 616

Text #9

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.9031. Min Act: -0.0001

Data Index: 598499 (C4 (Web Text))

Max Activating Token Index: 945

Text #10

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.9772. Min Act: -0.0001

Data Index: 431940 (C4 (Web Text))

Max Activating Token Index: 980

Text #11

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.8392. Min Act: -0.0001

Data Index: 756042 (C4 (Web Text))

Max Activating Token Index: 580

Text #12

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.8137. Min Act: -0.0001

Data Index: 1270500 (C4 (Web Text))

Max Activating Token Index: 466

Text #13

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.9035. Min Act: -0.0001

Data Index: 411221 (C4 (Web Text))

Max Activating Token Index: 575

Text #14

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.8892. Min Act: -0.0001

Data Index: 837784 (C4 (Web Text))

Max Activating Token Index: 142

Text #15

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.8678. Min Act: -0.0001

Data Index: 556849 (C4 (Web Text))

Max Activating Token Index: 524

Text #16

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.8600. Min Act: -0.0001

Data Index: 643605 (C4 (Web Text))

Max Activating Token Index: 228

Text #17

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.7247. Min Act: -0.0001

Data Index: 302223 (C4 (Web Text))

Max Activating Token Index: 957

Text #18

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.7199. Min Act: -0.0001

Data Index: 733795 (C4 (Web Text))

Max Activating Token Index: 932

Text #19

Max Range: 3.6543. Min Range: -3.6543

Max Act: 1.7732. Min Act: -0.0001

Data Index: 439997 (C4 (Web Text))

Max Activating Token Index: 448