Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1665 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0789. Min Range: -3.0789

Max Act: 3.0789. Min Act: -0.0000

Data Index: 1127371 (C4 (Web Text))

Max Activating Token Index: 891

Text #1

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.6828. Min Act: -0.0000

Data Index: 1356583 (C4 (Web Text))

Max Activating Token Index: 372

Text #2

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.6281. Min Act: -0.0000

Data Index: 965582 (C4 (Web Text))

Max Activating Token Index: 931

Text #3

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.9438. Min Act: -0.0000

Data Index: 662537 (C4 (Web Text))

Max Activating Token Index: 935

Text #4

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.9344. Min Act: -0.0000

Data Index: 1085103 (C4 (Web Text))

Max Activating Token Index: 759

Text #5

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.9221. Min Act: -0.0000

Data Index: 785486 (C4 (Web Text))

Max Activating Token Index: 658

Text #6

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.3283. Min Act: -0.0000

Data Index: 1094011 (C4 (Web Text))

Max Activating Token Index: 495

Text #7

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.1615. Min Act: -0.0000

Data Index: 12400 (C4 (Web Text))

Max Activating Token Index: 173

Text #8

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.0834. Min Act: -0.0000

Data Index: 277634 (C4 (Web Text))

Max Activating Token Index: 690

Text #9

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.1856. Min Act: -0.0000

Data Index: 273699 (C4 (Web Text))

Max Activating Token Index: 790

Text #10

Max Range: 3.0789. Min Range: -3.0789

Max Act: 2.0005. Min Act: -0.0000

Data Index: 1233614 (C4 (Web Text))

Max Activating Token Index: 620

Text #11

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.8691. Min Act: -0.0000

Data Index: 1099543 (C4 (Web Text))

Max Activating Token Index: 698

Text #12

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.8808. Min Act: -0.0000

Data Index: 391350 (C4 (Web Text))

Max Activating Token Index: 498

Text #13

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.7325. Min Act: -0.0000

Data Index: 1270905 (C4 (Web Text))

Max Activating Token Index: 968

Text #14

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.8012. Min Act: -0.0000

Data Index: 9943 (C4 (Web Text))

Max Activating Token Index: 373

Text #15

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.7258. Min Act: -0.0000

Data Index: 1309683 (C4 (Web Text))

Max Activating Token Index: 101

Text #16

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.6924. Min Act: -0.0000

Data Index: 796531 (C4 (Web Text))

Max Activating Token Index: 649

Text #17

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.6696. Min Act: -0.0000

Data Index: 1052247 (C4 (Web Text))

Max Activating Token Index: 239

Text #18

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.6449. Min Act: -0.0000

Data Index: 707995 (C4 (Web Text))

Max Activating Token Index: 978

Text #19

Max Range: 3.0789. Min Range: -3.0789

Max Act: 1.5345. Min Act: -0.0000

Data Index: 110676 (C4 (Web Text))

Max Activating Token Index: 337