Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4769 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.6820. Min Act: -0.0000

Data Index: 1222979 (C4 (Web Text))

Max Activating Token Index: 954

Text #1

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.5588. Min Act: -0.0000

Data Index: 584562 (C4 (Web Text))

Max Activating Token Index: 304

Text #2

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.5697. Min Act: -0.0000

Data Index: 715805 (C4 (Web Text))

Max Activating Token Index: 522

Text #3

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.4515. Min Act: -0.0000

Data Index: 1033858 (C4 (Web Text))

Max Activating Token Index: 117

Text #4

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.3128. Min Act: -0.0000

Data Index: 989410 (C4 (Web Text))

Max Activating Token Index: 556

Text #5

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.1611. Min Act: -0.0000

Data Index: 649632 (C4 (Web Text))

Max Activating Token Index: 602

Text #6

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.1699. Min Act: -0.0000

Data Index: 751094 (C4 (Web Text))

Max Activating Token Index: 172

Text #7

Max Range: 2.6820. Min Range: -2.6820

Max Act: 2.2364. Min Act: -0.0000

Data Index: 729418 (C4 (Web Text))

Max Activating Token Index: 621

Text #8

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.9466. Min Act: -0.0000

Data Index: 529315 (C4 (Web Text))

Max Activating Token Index: 860

Text #9

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.7813. Min Act: -0.0000

Data Index: 578782 (C4 (Web Text))

Max Activating Token Index: 529

Text #10

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.6763. Min Act: -0.0000

Data Index: 925561 (C4 (Web Text))

Max Activating Token Index: 343

Text #11

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.7711. Min Act: -0.0000

Data Index: 448808 (C4 (Web Text))

Max Activating Token Index: 988

Text #12

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.6572. Min Act: -0.0000

Data Index: 1204592 (C4 (Web Text))

Max Activating Token Index: 26

Text #13

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.8175. Min Act: -0.0000

Data Index: 823985 (C4 (Web Text))

Max Activating Token Index: 386

Text #14

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.6788. Min Act: -0.0000

Data Index: 257145 (C4 (Web Text))

Max Activating Token Index: 40

Text #15

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.7337. Min Act: -0.0000

Data Index: 1166731 (C4 (Web Text))

Max Activating Token Index: 697

Text #16

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.4757. Min Act: -0.0000

Data Index: 1216178 (C4 (Web Text))

Max Activating Token Index: 867

Text #17

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.6331. Min Act: -0.0000

Data Index: 1213552 (C4 (Web Text))

Max Activating Token Index: 533

Text #18

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.6212. Min Act: -0.0000

Data Index: 566315 (C4 (Web Text))

Max Activating Token Index: 284

Text #19

Max Range: 2.6820. Min Range: -2.6820

Max Act: 1.5941. Min Act: -0.0000

Data Index: 335716 (C4 (Web Text))

Max Activating Token Index: 978