Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2290 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3607. Min Act: -0.0001

Data Index: 1566282 (Python Code)

Max Activating Token Index: 37

Text #1

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3409. Min Act: -0.0001

Data Index: 1485973 (Python Code)

Max Activating Token Index: 53

Text #2

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3501. Min Act: -0.0001

Data Index: 1464779 (Python Code)

Max Activating Token Index: 183

Text #3

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3349. Min Act: -0.0001

Data Index: 1427858 (Python Code)

Max Activating Token Index: 82

Text #4

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3423. Min Act: -0.0001

Data Index: 1574239 (Python Code)

Max Activating Token Index: 43

Text #5

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3367. Min Act: -0.0001

Data Index: 1446395 (Python Code)

Max Activating Token Index: 702

Text #6

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2999. Min Act: -0.0001

Data Index: 1463877 (Python Code)

Max Activating Token Index: 158

Text #7

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3096. Min Act: -0.0001

Data Index: 1573119 (Python Code)

Max Activating Token Index: 74

Text #8

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3051. Min Act: -0.0001

Data Index: 1592736 (Python Code)

Max Activating Token Index: 48

Text #9

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3131. Min Act: -0.0001

Data Index: 1549341 (Python Code)

Max Activating Token Index: 486

Text #10

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.3144. Min Act: -0.0001

Data Index: 1386511 (Python Code)

Max Activating Token Index: 49

Text #11

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2922. Min Act: -0.0001

Data Index: 1478783 (Python Code)

Max Activating Token Index: 396

Text #12

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2990. Min Act: -0.0001

Data Index: 1467673 (Python Code)

Max Activating Token Index: 56

Text #13

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2970. Min Act: -0.0000

Data Index: 1468754 (Python Code)

Max Activating Token Index: 414

Text #14

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2851. Min Act: -0.0001

Data Index: 1387802 (Python Code)

Max Activating Token Index: 223

Text #15

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2819. Min Act: -0.0000

Data Index: 1377762 (Python Code)

Max Activating Token Index: 511

Text #16

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2826. Min Act: -0.0001

Data Index: 1526348 (Python Code)

Max Activating Token Index: 51

Text #17

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2802. Min Act: -0.0001

Data Index: 1429645 (Python Code)

Max Activating Token Index: 366

Text #18

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2803. Min Act: -0.0001

Data Index: 1492212 (Python Code)

Max Activating Token Index: 578

Text #19

Max Range: 0.3607. Min Range: -0.3607

Max Act: 0.2880. Min Act: -0.0001

Data Index: 1494253 (Python Code)

Max Activating Token Index: 334