Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4072 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.7520. Min Act: -0.0000

Data Index: 195932 (C4 (Web Text))

Max Activating Token Index: 153

Text #1

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.6748. Min Act: -0.0000

Data Index: 837649 (C4 (Web Text))

Max Activating Token Index: 960

Text #2

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.6517. Min Act: -0.0000

Data Index: 1338049 (C4 (Web Text))

Max Activating Token Index: 345

Text #3

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.6964. Min Act: -0.0000

Data Index: 1272103 (C4 (Web Text))

Max Activating Token Index: 520

Text #4

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5263. Min Act: -0.0000

Data Index: 1348786 (C4 (Web Text))

Max Activating Token Index: 1010

Text #5

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5591. Min Act: -0.0000

Data Index: 1006132 (C4 (Web Text))

Max Activating Token Index: 577

Text #6

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5406. Min Act: -0.0000

Data Index: 268776 (C4 (Web Text))

Max Activating Token Index: 490

Text #7

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5378. Min Act: -0.0000

Data Index: 458847 (C4 (Web Text))

Max Activating Token Index: 763

Text #8

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5518. Min Act: -0.0000

Data Index: 1022783 (C4 (Web Text))

Max Activating Token Index: 900

Text #9

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5096. Min Act: -0.0000

Data Index: 1148277 (C4 (Web Text))

Max Activating Token Index: 103

Text #10

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5445. Min Act: -0.0000

Data Index: 1192492 (C4 (Web Text))

Max Activating Token Index: 733

Text #11

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5018. Min Act: -0.0000

Data Index: 1159756 (C4 (Web Text))

Max Activating Token Index: 540

Text #12

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.4782. Min Act: -0.0000

Data Index: 305646 (C4 (Web Text))

Max Activating Token Index: 694

Text #13

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.4862. Min Act: -0.0000

Data Index: 1322402 (C4 (Web Text))

Max Activating Token Index: 261

Text #14

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.4867. Min Act: -0.0000

Data Index: 4439 (C4 (Web Text))

Max Activating Token Index: 62

Text #15

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.5131. Min Act: -0.0000

Data Index: 320363 (C4 (Web Text))

Max Activating Token Index: 599

Text #16

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.4580. Min Act: -0.0000

Data Index: 913777 (C4 (Web Text))

Max Activating Token Index: 890

Text #17

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.4190. Min Act: -0.0000

Data Index: 178252 (C4 (Web Text))

Max Activating Token Index: 441

Text #18

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.4768. Min Act: -0.0000

Data Index: 144721 (C4 (Web Text))

Max Activating Token Index: 980

Text #19

Max Range: 0.7520. Min Range: -0.7520

Max Act: 0.3931. Min Act: -0.0000

Data Index: 1235712 (C4 (Web Text))

Max Activating Token Index: 836