Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4519 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.4500. Min Act: -0.0000

Data Index: 1502600 (Python Code)

Max Activating Token Index: 612

Text #1

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3888. Min Act: -0.0000

Data Index: 1405075 (Python Code)

Max Activating Token Index: 941

Text #2

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3481. Min Act: -0.0000

Data Index: 1423749 (Python Code)

Max Activating Token Index: 935

Text #3

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3710. Min Act: -0.0000

Data Index: 1317957 (C4 (Web Text))

Max Activating Token Index: 20

Text #4

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3568. Min Act: -0.0000

Data Index: 1558455 (Python Code)

Max Activating Token Index: 127

Text #5

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3470. Min Act: -0.0000

Data Index: 1541679 (Python Code)

Max Activating Token Index: 384

Text #6

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3527. Min Act: -0.0000

Data Index: 1645399 (Python Code)

Max Activating Token Index: 256

Text #7

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3386. Min Act: -0.0000

Data Index: 1531768 (Python Code)

Max Activating Token Index: 958

Text #8

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3261. Min Act: -0.0000

Data Index: 1604807 (Python Code)

Max Activating Token Index: 250

Text #9

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3332. Min Act: 0.0000

Data Index: 1602299 (Python Code)

Max Activating Token Index: 763

Text #10

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3213. Min Act: -0.0000

Data Index: 925098 (C4 (Web Text))

Max Activating Token Index: 909

Text #11

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3301. Min Act: -0.0000

Data Index: 404455 (C4 (Web Text))

Max Activating Token Index: 516

Text #12

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3011. Min Act: 0.0000

Data Index: 1572229 (Python Code)

Max Activating Token Index: 153

Text #13

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3122. Min Act: -0.0000

Data Index: 1405442 (Python Code)

Max Activating Token Index: 446

Text #14

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3219. Min Act: -0.0000

Data Index: 311806 (C4 (Web Text))

Max Activating Token Index: 691

Text #15

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.2999. Min Act: -0.0000

Data Index: 1618511 (Python Code)

Max Activating Token Index: 645

Text #16

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.3026. Min Act: -0.0000

Data Index: 1458139 (Python Code)

Max Activating Token Index: 476

Text #17

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.2873. Min Act: -0.0000

Data Index: 1653337 (Python Code)

Max Activating Token Index: 161

Text #18

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.2954. Min Act: -0.0000

Data Index: 1571004 (Python Code)

Max Activating Token Index: 907

Text #19

Max Range: 0.4500. Min Range: -0.4500

Max Act: 0.2954. Min Act: -0.0000

Data Index: 1528690 (Python Code)

Max Activating Token Index: 137