Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3731 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.8253. Min Act: -0.0000

Data Index: 250049 (C4 (Web Text))

Max Activating Token Index: 392

Text #1

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.8588. Min Act: -0.0000

Data Index: 205712 (C4 (Web Text))

Max Activating Token Index: 955

Text #2

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.8454. Min Act: -0.0000

Data Index: 1149735 (C4 (Web Text))

Max Activating Token Index: 580

Text #3

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.8196. Min Act: -0.0000

Data Index: 1003007 (C4 (Web Text))

Max Activating Token Index: 556

Text #4

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.8124. Min Act: -0.0000

Data Index: 503385 (C4 (Web Text))

Max Activating Token Index: 200

Text #5

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.8028. Min Act: -0.0000

Data Index: 958820 (C4 (Web Text))

Max Activating Token Index: 13

Text #6

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7681. Min Act: -0.0000

Data Index: 1258867 (C4 (Web Text))

Max Activating Token Index: 311

Text #7

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7356. Min Act: -0.0000

Data Index: 983340 (C4 (Web Text))

Max Activating Token Index: 589

Text #8

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7810. Min Act: -0.0000

Data Index: 786322 (C4 (Web Text))

Max Activating Token Index: 981

Text #9

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7731. Min Act: -0.0000

Data Index: 17231 (C4 (Web Text))

Max Activating Token Index: 741

Text #10

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7539. Min Act: -0.0000

Data Index: 1079958 (C4 (Web Text))

Max Activating Token Index: 429

Text #11

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7262. Min Act: -0.0000

Data Index: 520894 (C4 (Web Text))

Max Activating Token Index: 30

Text #12

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.6765. Min Act: -0.0000

Data Index: 79854 (C4 (Web Text))

Max Activating Token Index: 182

Text #13

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7182. Min Act: -0.0000

Data Index: 360446 (C4 (Web Text))

Max Activating Token Index: 508

Text #14

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7165. Min Act: -0.0000

Data Index: 985994 (C4 (Web Text))

Max Activating Token Index: 491

Text #15

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7795. Min Act: -0.0000

Data Index: 1304996 (C4 (Web Text))

Max Activating Token Index: 1007

Text #16

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.6905. Min Act: -0.0000

Data Index: 698367 (C4 (Web Text))

Max Activating Token Index: 527

Text #17

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.6690. Min Act: -0.0000

Data Index: 1033997 (C4 (Web Text))

Max Activating Token Index: 561

Text #18

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.7113. Min Act: -0.0000

Data Index: 1199605 (C4 (Web Text))

Max Activating Token Index: 257

Text #19

Max Range: 0.8588. Min Range: -0.8588

Max Act: 0.6790. Min Act: -0.0000

Data Index: 949009 (C4 (Web Text))

Max Activating Token Index: 814