Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 275 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.7953. Min Act: -0.0000

Data Index: 382083 (C4 (Web Text))

Max Activating Token Index: 123

Text #1

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.4863. Min Act: -0.0000

Data Index: 165045 (C4 (Web Text))

Max Activating Token Index: 578

Text #2

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.1479. Min Act: -0.0000

Data Index: 217015 (C4 (Web Text))

Max Activating Token Index: 665

Text #3

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.2674. Min Act: -0.0000

Data Index: 628735 (C4 (Web Text))

Max Activating Token Index: 683

Text #4

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.1550. Min Act: -0.0000

Data Index: 897942 (C4 (Web Text))

Max Activating Token Index: 974

Text #5

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.1079. Min Act: -0.0000

Data Index: 1278982 (C4 (Web Text))

Max Activating Token Index: 466

Text #6

Max Range: 3.7953. Min Range: -3.7953

Max Act: 3.0186. Min Act: -0.0000

Data Index: 175379 (C4 (Web Text))

Max Activating Token Index: 481

Text #7

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.9804. Min Act: -0.0000

Data Index: 254240 (C4 (Web Text))

Max Activating Token Index: 720

Text #8

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.8565. Min Act: -0.0000

Data Index: 136602 (C4 (Web Text))

Max Activating Token Index: 182

Text #9

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.8507. Min Act: -0.0000

Data Index: 471464 (C4 (Web Text))

Max Activating Token Index: 759

Text #10

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.9109. Min Act: -0.0000

Data Index: 1250352 (C4 (Web Text))

Max Activating Token Index: 64

Text #11

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.7341. Min Act: -0.0000

Data Index: 585342 (C4 (Web Text))

Max Activating Token Index: 979

Text #12

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.6896. Min Act: -0.0000

Data Index: 418594 (C4 (Web Text))

Max Activating Token Index: 615

Text #13

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.6092. Min Act: -0.0000

Data Index: 456476 (C4 (Web Text))

Max Activating Token Index: 348

Text #14

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.4937. Min Act: -0.0000

Data Index: 231754 (C4 (Web Text))

Max Activating Token Index: 207

Text #15

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.5869. Min Act: -0.0000

Data Index: 530521 (C4 (Web Text))

Max Activating Token Index: 208

Text #16

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.4028. Min Act: -0.0000

Data Index: 1298992 (C4 (Web Text))

Max Activating Token Index: 1000

Text #17

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.3763. Min Act: -0.0000

Data Index: 865361 (C4 (Web Text))

Max Activating Token Index: 25

Text #18

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.5170. Min Act: -0.0000

Data Index: 146450 (C4 (Web Text))

Max Activating Token Index: 715

Text #19

Max Range: 3.7953. Min Range: -3.7953

Max Act: 2.3698. Min Act: -0.0000

Data Index: 533217 (C4 (Web Text))

Max Activating Token Index: 992