Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5314 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1561. Min Range: -2.1561

Max Act: 2.1082. Min Act: -0.0000

Data Index: 769062 (C4 (Web Text))

Max Activating Token Index: 781

Text #1

Max Range: 2.1561. Min Range: -2.1561

Max Act: 2.1561. Min Act: -0.0000

Data Index: 303330 (C4 (Web Text))

Max Activating Token Index: 936

Text #2

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8669. Min Act: -0.0000

Data Index: 563501 (C4 (Web Text))

Max Activating Token Index: 1003

Text #3

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.9393. Min Act: -0.0000

Data Index: 137057 (C4 (Web Text))

Max Activating Token Index: 36

Text #4

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.9455. Min Act: -0.0000

Data Index: 1298201 (C4 (Web Text))

Max Activating Token Index: 967

Text #5

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8222. Min Act: -0.0000

Data Index: 638767 (C4 (Web Text))

Max Activating Token Index: 823

Text #6

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8922. Min Act: -0.0000

Data Index: 277970 (C4 (Web Text))

Max Activating Token Index: 754

Text #7

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.7676. Min Act: -0.0000

Data Index: 450339 (C4 (Web Text))

Max Activating Token Index: 240

Text #8

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8641. Min Act: -0.0000

Data Index: 607457 (C4 (Web Text))

Max Activating Token Index: 589

Text #9

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.9825. Min Act: -0.0000

Data Index: 1025487 (C4 (Web Text))

Max Activating Token Index: 990

Text #10

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8485. Min Act: -0.0000

Data Index: 231485 (C4 (Web Text))

Max Activating Token Index: 622

Text #11

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8536. Min Act: -0.0000

Data Index: 1358054 (C4 (Web Text))

Max Activating Token Index: 458

Text #12

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.7399. Min Act: -0.0000

Data Index: 1470187 (Python Code)

Max Activating Token Index: 66

Text #13

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8324. Min Act: -0.0000

Data Index: 606243 (C4 (Web Text))

Max Activating Token Index: 100

Text #14

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.7135. Min Act: -0.0000

Data Index: 263929 (C4 (Web Text))

Max Activating Token Index: 377

Text #15

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.7940. Min Act: -0.0000

Data Index: 1277124 (C4 (Web Text))

Max Activating Token Index: 128

Text #16

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.7923. Min Act: -0.0000

Data Index: 248180 (C4 (Web Text))

Max Activating Token Index: 895

Text #17

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.8816. Min Act: -0.0000

Data Index: 189625 (C4 (Web Text))

Max Activating Token Index: 84

Text #18

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.7780. Min Act: -0.0000

Data Index: 890101 (C4 (Web Text))

Max Activating Token Index: 120

Text #19

Max Range: 2.1561. Min Range: -2.1561

Max Act: 1.6759. Min Act: -0.0000

Data Index: 501088 (C4 (Web Text))

Max Activating Token Index: 867