Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 669 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.5297. Min Act: -0.0002

Data Index: 1410505 (Python Code)

Max Activating Token Index: 396

Text #1

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.5170. Min Act: -0.0002

Data Index: 1362489 (Python Code)

Max Activating Token Index: 769

Text #2

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.5168. Min Act: -0.0002

Data Index: 1637175 (Python Code)

Max Activating Token Index: 585

Text #3

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.4702. Min Act: -0.0002

Data Index: 1453164 (Python Code)

Max Activating Token Index: 205

Text #4

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.4276. Min Act: -0.0002

Data Index: 1371431 (Python Code)

Max Activating Token Index: 245

Text #5

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.4026. Min Act: -0.0002

Data Index: 1518342 (Python Code)

Max Activating Token Index: 653

Text #6

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.4387. Min Act: -0.0002

Data Index: 1383203 (Python Code)

Max Activating Token Index: 636

Text #7

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3703. Min Act: -0.0002

Data Index: 1428650 (Python Code)

Max Activating Token Index: 355

Text #8

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3590. Min Act: -0.0002

Data Index: 1439075 (Python Code)

Max Activating Token Index: 383

Text #9

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3534. Min Act: -0.0002

Data Index: 1590019 (Python Code)

Max Activating Token Index: 753

Text #10

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3489. Min Act: -0.0002

Data Index: 1612397 (Python Code)

Max Activating Token Index: 693

Text #11

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.4170. Min Act: -0.0002

Data Index: 1371013 (Python Code)

Max Activating Token Index: 760

Text #12

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3347. Min Act: -0.0002

Data Index: 1520241 (Python Code)

Max Activating Token Index: 1019

Text #13

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3248. Min Act: -0.0002

Data Index: 1601534 (Python Code)

Max Activating Token Index: 516

Text #14

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3861. Min Act: -0.0002

Data Index: 1498139 (Python Code)

Max Activating Token Index: 668

Text #15

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3612. Min Act: -0.0002

Data Index: 1436627 (Python Code)

Max Activating Token Index: 752

Text #16

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3158. Min Act: -0.0002

Data Index: 1435083 (Python Code)

Max Activating Token Index: 542

Text #17

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3069. Min Act: -0.0002

Data Index: 1457767 (Python Code)

Max Activating Token Index: 920

Text #18

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3020. Min Act: -0.0002

Data Index: 1539262 (Python Code)

Max Activating Token Index: 676

Text #19

Max Range: 1.5297. Min Range: -1.5297

Max Act: 1.3796. Min Act: -0.0002

Data Index: 1416441 (Python Code)

Max Activating Token Index: 619