Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3194 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1674. Min Range: -1.1674

Max Act: 1.1674. Min Act: -0.0000

Data Index: 102254 (C4 (Web Text))

Max Activating Token Index: 871

Text #1

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.9434. Min Act: -0.0000

Data Index: 1252333 (C4 (Web Text))

Max Activating Token Index: 232

Text #2

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.8283. Min Act: -0.0000

Data Index: 511869 (C4 (Web Text))

Max Activating Token Index: 131

Text #3

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.7642. Min Act: -0.0000

Data Index: 878931 (C4 (Web Text))

Max Activating Token Index: 959

Text #4

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.7169. Min Act: -0.0000

Data Index: 1184232 (C4 (Web Text))

Max Activating Token Index: 693

Text #5

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6813. Min Act: -0.0000

Data Index: 952131 (C4 (Web Text))

Max Activating Token Index: 398

Text #6

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.7249. Min Act: -0.0000

Data Index: 437059 (C4 (Web Text))

Max Activating Token Index: 936

Text #7

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.7111. Min Act: -0.0000

Data Index: 919947 (C4 (Web Text))

Max Activating Token Index: 969

Text #8

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6447. Min Act: -0.0000

Data Index: 695470 (C4 (Web Text))

Max Activating Token Index: 767

Text #9

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6930. Min Act: -0.0000

Data Index: 1064118 (C4 (Web Text))

Max Activating Token Index: 550

Text #10

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6878. Min Act: -0.0000

Data Index: 2770 (C4 (Web Text))

Max Activating Token Index: 17

Text #11

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6574. Min Act: -0.0000

Data Index: 191480 (C4 (Web Text))

Max Activating Token Index: 399

Text #12

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6551. Min Act: -0.0000

Data Index: 757690 (C4 (Web Text))

Max Activating Token Index: 285

Text #13

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6413. Min Act: -0.0000

Data Index: 465910 (C4 (Web Text))

Max Activating Token Index: 722

Text #14

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6444. Min Act: -0.0000

Data Index: 604554 (C4 (Web Text))

Max Activating Token Index: 815

Text #15

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6691. Min Act: -0.0000

Data Index: 1224308 (C4 (Web Text))

Max Activating Token Index: 981

Text #16

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6466. Min Act: -0.0000

Data Index: 39098 (C4 (Web Text))

Max Activating Token Index: 925

Text #17

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6596. Min Act: -0.0000

Data Index: 1306938 (C4 (Web Text))

Max Activating Token Index: 83

Text #18

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6320. Min Act: -0.0000

Data Index: 756636 (C4 (Web Text))

Max Activating Token Index: 734

Text #19

Max Range: 1.1674. Min Range: -1.1674

Max Act: 0.6356. Min Act: -0.0000

Data Index: 1228420 (C4 (Web Text))

Max Activating Token Index: 508