Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1261 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.2998. Min Range: -5.2998

Max Act: 5.0750. Min Act: -0.0000

Data Index: 1193940 (C4 (Web Text))

Max Activating Token Index: 900

Text #1

Max Range: 5.2998. Min Range: -5.2998

Max Act: 5.0314. Min Act: -0.0000

Data Index: 1177945 (C4 (Web Text))

Max Activating Token Index: 866

Text #2

Max Range: 5.2998. Min Range: -5.2998

Max Act: 5.2712. Min Act: -0.0000

Data Index: 367025 (C4 (Web Text))

Max Activating Token Index: 974

Text #3

Max Range: 5.2998. Min Range: -5.2998

Max Act: 5.2998. Min Act: -0.0000

Data Index: 92001 (C4 (Web Text))

Max Activating Token Index: 757

Text #4

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.9971. Min Act: -0.0000

Data Index: 371785 (C4 (Web Text))

Max Activating Token Index: 549

Text #5

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.9633. Min Act: -0.0000

Data Index: 377317 (C4 (Web Text))

Max Activating Token Index: 374

Text #6

Max Range: 5.2998. Min Range: -5.2998

Max Act: 5.0654. Min Act: -0.0000

Data Index: 320694 (C4 (Web Text))

Max Activating Token Index: 756

Text #7

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.9812. Min Act: -0.0000

Data Index: 180897 (C4 (Web Text))

Max Activating Token Index: 461

Text #8

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.9410. Min Act: -0.0000

Data Index: 317179 (C4 (Web Text))

Max Activating Token Index: 112

Text #9

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.7626. Min Act: -0.0000

Data Index: 1342484 (C4 (Web Text))

Max Activating Token Index: 341

Text #10

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.7108. Min Act: -0.0000

Data Index: 884856 (C4 (Web Text))

Max Activating Token Index: 22

Text #11

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.2971. Min Act: -0.0000

Data Index: 1154503 (C4 (Web Text))

Max Activating Token Index: 131

Text #12

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.4560. Min Act: -0.0000

Data Index: 1358234 (C4 (Web Text))

Max Activating Token Index: 586

Text #13

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.6048. Min Act: -0.0000

Data Index: 409882 (C4 (Web Text))

Max Activating Token Index: 157

Text #14

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.5451. Min Act: -0.0000

Data Index: 1249746 (C4 (Web Text))

Max Activating Token Index: 500

Text #15

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.3515. Min Act: -0.0000

Data Index: 155079 (C4 (Web Text))

Max Activating Token Index: 65

Text #16

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.5113. Min Act: -0.0000

Data Index: 410588 (C4 (Web Text))

Max Activating Token Index: 199

Text #17

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.2702. Min Act: -0.0000

Data Index: 354728 (C4 (Web Text))

Max Activating Token Index: 419

Text #18

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.4282. Min Act: -0.0000

Data Index: 781157 (C4 (Web Text))

Max Activating Token Index: 831

Text #19

Max Range: 5.2998. Min Range: -5.2998

Max Act: 4.3999. Min Act: -0.0000

Data Index: 797808 (C4 (Web Text))

Max Activating Token Index: 846