Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 53 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.9128. Min Act: -0.0000

Data Index: 96986 (C4 (Web Text))

Max Activating Token Index: 541

Text #1

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.8664. Min Act: -0.0000

Data Index: 59826 (C4 (Web Text))

Max Activating Token Index: 709

Text #2

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.8262. Min Act: -0.0000

Data Index: 359337 (C4 (Web Text))

Max Activating Token Index: 135

Text #3

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.7900. Min Act: -0.0000

Data Index: 108998 (C4 (Web Text))

Max Activating Token Index: 452

Text #4

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.8120. Min Act: -0.0000

Data Index: 809288 (C4 (Web Text))

Max Activating Token Index: 194

Text #5

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.7750. Min Act: -0.0000

Data Index: 814020 (C4 (Web Text))

Max Activating Token Index: 900

Text #6

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.7101. Min Act: -0.0000

Data Index: 1205450 (C4 (Web Text))

Max Activating Token Index: 42

Text #7

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.7122. Min Act: -0.0000

Data Index: 1250109 (C4 (Web Text))

Max Activating Token Index: 243

Text #8

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.7895. Min Act: -0.0000

Data Index: 366702 (C4 (Web Text))

Max Activating Token Index: 560

Text #9

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6696. Min Act: -0.0000

Data Index: 369794 (C4 (Web Text))

Max Activating Token Index: 491

Text #10

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.7915. Min Act: -0.0000

Data Index: 1088901 (C4 (Web Text))

Max Activating Token Index: 801

Text #11

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6457. Min Act: -0.0000

Data Index: 87369 (C4 (Web Text))

Max Activating Token Index: 479

Text #12

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6888. Min Act: -0.0000

Data Index: 1053024 (C4 (Web Text))

Max Activating Token Index: 10

Text #13

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.5173. Min Act: -0.0000

Data Index: 756637 (C4 (Web Text))

Max Activating Token Index: 1016

Text #14

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6220. Min Act: -0.0000

Data Index: 1069094 (C4 (Web Text))

Max Activating Token Index: 860

Text #15

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6619. Min Act: -0.0000

Data Index: 460671 (C4 (Web Text))

Max Activating Token Index: 11

Text #16

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.5991. Min Act: -0.0000

Data Index: 385574 (C4 (Web Text))

Max Activating Token Index: 140

Text #17

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6602. Min Act: -0.0000

Data Index: 715965 (C4 (Web Text))

Max Activating Token Index: 1021

Text #18

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6672. Min Act: -0.0000

Data Index: 1354453 (C4 (Web Text))

Max Activating Token Index: 765

Text #19

Max Range: 0.9128. Min Range: -0.9128

Max Act: 0.6484. Min Act: -0.0000

Data Index: 402958 (C4 (Web Text))

Max Activating Token Index: 562