Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3735 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2761. Min Act: -0.0000

Data Index: 1321972 (C4 (Web Text))

Max Activating Token Index: 398

Text #1

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2482. Min Act: -0.0000

Data Index: 996744 (C4 (Web Text))

Max Activating Token Index: 509

Text #2

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2251. Min Act: -0.0000

Data Index: 30749 (C4 (Web Text))

Max Activating Token Index: 450

Text #3

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2251. Min Act: -0.0000

Data Index: 885335 (C4 (Web Text))

Max Activating Token Index: 60

Text #4

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2146. Min Act: -0.0000

Data Index: 776532 (C4 (Web Text))

Max Activating Token Index: 548

Text #5

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2079. Min Act: -0.0000

Data Index: 158617 (C4 (Web Text))

Max Activating Token Index: 572

Text #6

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1966. Min Act: -0.0000

Data Index: 360015 (C4 (Web Text))

Max Activating Token Index: 309

Text #7

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1921. Min Act: -0.0000

Data Index: 883131 (C4 (Web Text))

Max Activating Token Index: 888

Text #8

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1944. Min Act: -0.0000

Data Index: 884248 (C4 (Web Text))

Max Activating Token Index: 629

Text #9

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1987. Min Act: -0.0000

Data Index: 34623 (C4 (Web Text))

Max Activating Token Index: 548

Text #10

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.2065. Min Act: -0.0000

Data Index: 1152963 (C4 (Web Text))

Max Activating Token Index: 218

Text #11

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1974. Min Act: -0.0000

Data Index: 788104 (C4 (Web Text))

Max Activating Token Index: 827

Text #12

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1872. Min Act: -0.0000

Data Index: 698394 (C4 (Web Text))

Max Activating Token Index: 776

Text #13

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1784. Min Act: -0.0000

Data Index: 1280200 (C4 (Web Text))

Max Activating Token Index: 638

Text #14

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1920. Min Act: -0.0000

Data Index: 1226816 (C4 (Web Text))

Max Activating Token Index: 19

Text #15

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1955. Min Act: -0.0000

Data Index: 669380 (C4 (Web Text))

Max Activating Token Index: 458

Text #16

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1910. Min Act: -0.0000

Data Index: 928949 (C4 (Web Text))

Max Activating Token Index: 248

Text #17

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1789. Min Act: -0.0000

Data Index: 955011 (C4 (Web Text))

Max Activating Token Index: 488

Text #18

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1862. Min Act: -0.0000

Data Index: 504675 (C4 (Web Text))

Max Activating Token Index: 837

Text #19

Max Range: 0.2761. Min Range: -0.2761

Max Act: 0.1923. Min Act: -0.0000

Data Index: 597682 (C4 (Web Text))

Max Activating Token Index: 210