Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3918 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.5459. Min Act: -0.0000

Data Index: 1537042 (Python Code)

Max Activating Token Index: 416

Text #1

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.2197. Min Act: -0.0000

Data Index: 1610941 (Python Code)

Max Activating Token Index: 965

Text #2

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.4930. Min Act: -0.0000

Data Index: 1383984 (Python Code)

Max Activating Token Index: 199

Text #3

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.1967. Min Act: -0.0000

Data Index: 1404448 (Python Code)

Max Activating Token Index: 321

Text #4

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.1643. Min Act: -0.0000

Data Index: 1525433 (Python Code)

Max Activating Token Index: 574

Text #5

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.1204. Min Act: -0.0000

Data Index: 1526406 (Python Code)

Max Activating Token Index: 453

Text #6

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.1127. Min Act: -0.0000

Data Index: 1443354 (Python Code)

Max Activating Token Index: 734

Text #7

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.8363. Min Act: -0.0000

Data Index: 1491055 (Python Code)

Max Activating Token Index: 145

Text #8

Max Range: 3.5459. Min Range: -3.5459

Max Act: 3.0907. Min Act: -0.0000

Data Index: 1636441 (Python Code)

Max Activating Token Index: 111

Text #9

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.7936. Min Act: -0.0000

Data Index: 1617727 (Python Code)

Max Activating Token Index: 498

Text #10

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.7766. Min Act: -0.0000

Data Index: 1499188 (Python Code)

Max Activating Token Index: 86

Text #11

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.8666. Min Act: -0.0000

Data Index: 1361065 (Python Code)

Max Activating Token Index: 114

Text #12

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.8380. Min Act: -0.0000

Data Index: 1444501 (Python Code)

Max Activating Token Index: 387

Text #13

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.7319. Min Act: -0.0000

Data Index: 1560216 (Python Code)

Max Activating Token Index: 74

Text #14

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.7150. Min Act: -0.0000

Data Index: 1636404 (Python Code)

Max Activating Token Index: 92

Text #15

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.6954. Min Act: -0.0000

Data Index: 1517107 (Python Code)

Max Activating Token Index: 463

Text #16

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.6725. Min Act: -0.0000

Data Index: 1552703 (Python Code)

Max Activating Token Index: 889

Text #17

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.6617. Min Act: -0.0000

Data Index: 1531976 (Python Code)

Max Activating Token Index: 154

Text #18

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.5467. Min Act: -0.0000

Data Index: 1463382 (Python Code)

Max Activating Token Index: 810

Text #19

Max Range: 3.5459. Min Range: -3.5459

Max Act: 2.6626. Min Act: -0.0000

Data Index: 1517832 (Python Code)

Max Activating Token Index: 865