Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1109 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.8338. Min Range: -4.8338

Max Act: 4.8338. Min Act: -0.0000

Data Index: 498759 (C4 (Web Text))

Max Activating Token Index: 953

Text #1

Max Range: 4.8338. Min Range: -4.8338

Max Act: 4.6922. Min Act: -0.0000

Data Index: 1068796 (C4 (Web Text))

Max Activating Token Index: 507

Text #2

Max Range: 4.8338. Min Range: -4.8338

Max Act: 4.5002. Min Act: -0.0000

Data Index: 1232420 (C4 (Web Text))

Max Activating Token Index: 176

Text #3

Max Range: 4.8338. Min Range: -4.8338

Max Act: 4.2074. Min Act: -0.0000

Data Index: 976634 (C4 (Web Text))

Max Activating Token Index: 834

Text #4

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.8579. Min Act: -0.0000

Data Index: 1197121 (C4 (Web Text))

Max Activating Token Index: 462

Text #5

Max Range: 4.8338. Min Range: -4.8338

Max Act: 4.0118. Min Act: -0.0000

Data Index: 189243 (C4 (Web Text))

Max Activating Token Index: 670

Text #6

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.5846. Min Act: -0.0000

Data Index: 925310 (C4 (Web Text))

Max Activating Token Index: 853

Text #7

Max Range: 4.8338. Min Range: -4.8338

Max Act: 4.0416. Min Act: -0.0000

Data Index: 1015045 (C4 (Web Text))

Max Activating Token Index: 778

Text #8

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.8545. Min Act: -0.0000

Data Index: 1172029 (C4 (Web Text))

Max Activating Token Index: 697

Text #9

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.9436. Min Act: -0.0000

Data Index: 411134 (C4 (Web Text))

Max Activating Token Index: 534

Text #10

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.5971. Min Act: -0.0000

Data Index: 1239962 (C4 (Web Text))

Max Activating Token Index: 958

Text #11

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.7010. Min Act: -0.0000

Data Index: 722364 (C4 (Web Text))

Max Activating Token Index: 874

Text #12

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.4692. Min Act: -0.0000

Data Index: 155864 (C4 (Web Text))

Max Activating Token Index: 709

Text #13

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.5784. Min Act: -0.0000

Data Index: 806200 (C4 (Web Text))

Max Activating Token Index: 711

Text #14

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.5602. Min Act: -0.0000

Data Index: 860466 (C4 (Web Text))

Max Activating Token Index: 515

Text #15

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.5603. Min Act: -0.0000

Data Index: 1241079 (C4 (Web Text))

Max Activating Token Index: 865

Text #16

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.6087. Min Act: -0.0000

Data Index: 651922 (C4 (Web Text))

Max Activating Token Index: 1001

Text #17

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.2721. Min Act: -0.0000

Data Index: 649273 (C4 (Web Text))

Max Activating Token Index: 489

Text #18

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.4296. Min Act: -0.0000

Data Index: 134654 (C4 (Web Text))

Max Activating Token Index: 700

Text #19

Max Range: 4.8338. Min Range: -4.8338

Max Act: 3.5568. Min Act: -0.0000

Data Index: 1081265 (C4 (Web Text))

Max Activating Token Index: 742