Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2554 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1077. Min Range: -1.1077

Max Act: 1.1077. Min Act: -0.0000

Data Index: 1530454 (Python Code)

Max Activating Token Index: 133

Text #1

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.9335. Min Act: -0.0000

Data Index: 1541038 (Python Code)

Max Activating Token Index: 356

Text #2

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.9503. Min Act: -0.0000

Data Index: 1367240 (Python Code)

Max Activating Token Index: 748

Text #3

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8939. Min Act: -0.0000

Data Index: 819397 (C4 (Web Text))

Max Activating Token Index: 292

Text #4

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8448. Min Act: -0.0000

Data Index: 1387465 (Python Code)

Max Activating Token Index: 993

Text #5

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.9106. Min Act: -0.0000

Data Index: 1434463 (Python Code)

Max Activating Token Index: 730

Text #6

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8862. Min Act: -0.0000

Data Index: 1425919 (Python Code)

Max Activating Token Index: 904

Text #7

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8278. Min Act: -0.0000

Data Index: 1435402 (Python Code)

Max Activating Token Index: 201

Text #8

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8482. Min Act: -0.0000

Data Index: 1395929 (Python Code)

Max Activating Token Index: 721

Text #9

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8534. Min Act: -0.0000

Data Index: 1520589 (Python Code)

Max Activating Token Index: 1008

Text #10

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8307. Min Act: -0.0000

Data Index: 1583252 (Python Code)

Max Activating Token Index: 434

Text #11

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.7952. Min Act: -0.0000

Data Index: 1528404 (Python Code)

Max Activating Token Index: 253

Text #12

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8202. Min Act: -0.0000

Data Index: 1508531 (Python Code)

Max Activating Token Index: 480

Text #13

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8107. Min Act: -0.0000

Data Index: 1419050 (Python Code)

Max Activating Token Index: 516

Text #14

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.7976. Min Act: -0.0000

Data Index: 1557992 (Python Code)

Max Activating Token Index: 783

Text #15

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.8201. Min Act: -0.0000

Data Index: 1361461 (Python Code)

Max Activating Token Index: 224

Text #16

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.7476. Min Act: -0.0000

Data Index: 1382335 (Python Code)

Max Activating Token Index: 308

Text #17

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.7815. Min Act: -0.0000

Data Index: 1387929 (Python Code)

Max Activating Token Index: 64

Text #18

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.7966. Min Act: -0.0000

Data Index: 1399267 (Python Code)

Max Activating Token Index: 598

Text #19

Max Range: 1.1077. Min Range: -1.1077

Max Act: 0.7395. Min Act: -0.0000

Data Index: 1393571 (Python Code)

Max Activating Token Index: 337