Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2072 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.3646. Min Range: -3.3646

Max Act: 3.3646. Min Act: -0.0000

Data Index: 729458 (C4 (Web Text))

Max Activating Token Index: 91

Text #1

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.5732. Min Act: -0.0000

Data Index: 1213253 (C4 (Web Text))

Max Activating Token Index: 311

Text #2

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.2752. Min Act: -0.0000

Data Index: 231188 (C4 (Web Text))

Max Activating Token Index: 193

Text #3

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.1582. Min Act: -0.0000

Data Index: 345136 (C4 (Web Text))

Max Activating Token Index: 62

Text #4

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.1582. Min Act: -0.0000

Data Index: 679570 (C4 (Web Text))

Max Activating Token Index: 62

Text #5

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.2393. Min Act: -0.0000

Data Index: 46393 (C4 (Web Text))

Max Activating Token Index: 983

Text #6

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.2238. Min Act: -0.0000

Data Index: 799007 (C4 (Web Text))

Max Activating Token Index: 983

Text #7

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9803. Min Act: -0.0000

Data Index: 1195071 (C4 (Web Text))

Max Activating Token Index: 921

Text #8

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.0741. Min Act: -0.0000

Data Index: 682645 (C4 (Web Text))

Max Activating Token Index: 187

Text #9

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9365. Min Act: -0.0000

Data Index: 95935 (C4 (Web Text))

Max Activating Token Index: 861

Text #10

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.0161. Min Act: -0.0000

Data Index: 86805 (C4 (Web Text))

Max Activating Token Index: 406

Text #11

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.0127. Min Act: -0.0000

Data Index: 363121 (C4 (Web Text))

Max Activating Token Index: 730

Text #12

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9910. Min Act: -0.0000

Data Index: 992526 (C4 (Web Text))

Max Activating Token Index: 140

Text #13

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.0723. Min Act: -0.0000

Data Index: 1067934 (C4 (Web Text))

Max Activating Token Index: 842

Text #14

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9514. Min Act: -0.0000

Data Index: 18786 (C4 (Web Text))

Max Activating Token Index: 255

Text #15

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9555. Min Act: -0.0000

Data Index: 974371 (C4 (Web Text))

Max Activating Token Index: 872

Text #16

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.8547. Min Act: -0.0000

Data Index: 482521 (C4 (Web Text))

Max Activating Token Index: 843

Text #17

Max Range: 3.3646. Min Range: -3.3646

Max Act: 2.0470. Min Act: -0.0000

Data Index: 500316 (C4 (Web Text))

Max Activating Token Index: 227

Text #18

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9410. Min Act: -0.0000

Data Index: 792524 (C4 (Web Text))

Max Activating Token Index: 527

Text #19

Max Range: 3.3646. Min Range: -3.3646

Max Act: 1.9195. Min Act: -0.0000

Data Index: 1096948 (C4 (Web Text))

Max Activating Token Index: 650