Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5061 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.9350. Min Act: -0.0000

Data Index: 1445654 (Python Code)

Max Activating Token Index: 208

Text #1

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.5299. Min Act: -0.0000

Data Index: 1522277 (Python Code)

Max Activating Token Index: 623

Text #2

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.3731. Min Act: -0.0000

Data Index: 1432324 (Python Code)

Max Activating Token Index: 569

Text #3

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.3244. Min Act: -0.0000

Data Index: 1423967 (Python Code)

Max Activating Token Index: 186

Text #4

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.1265. Min Act: -0.0000

Data Index: 1637443 (Python Code)

Max Activating Token Index: 528

Text #5

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.3065. Min Act: -0.0000

Data Index: 1488118 (Python Code)

Max Activating Token Index: 794

Text #6

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.1484. Min Act: -0.0000

Data Index: 1558022 (Python Code)

Max Activating Token Index: 42

Text #7

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.9170. Min Act: -0.0000

Data Index: 1605345 (Python Code)

Max Activating Token Index: 769

Text #8

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.9832. Min Act: -0.0000

Data Index: 1447336 (Python Code)

Max Activating Token Index: 120

Text #9

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.9285. Min Act: -0.0000

Data Index: 1435824 (Python Code)

Max Activating Token Index: 587

Text #10

Max Range: 2.9350. Min Range: -2.9350

Max Act: 2.0702. Min Act: -0.0000

Data Index: 1456593 (Python Code)

Max Activating Token Index: 235

Text #11

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.9617. Min Act: -0.0000

Data Index: 1433759 (Python Code)

Max Activating Token Index: 496

Text #12

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.8817. Min Act: -0.0000

Data Index: 1570939 (Python Code)

Max Activating Token Index: 973

Text #13

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.7944. Min Act: -0.0000

Data Index: 1555445 (Python Code)

Max Activating Token Index: 427

Text #14

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.6460. Min Act: -0.0000

Data Index: 1405611 (Python Code)

Max Activating Token Index: 775

Text #15

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.6623. Min Act: -0.0000

Data Index: 1551180 (Python Code)

Max Activating Token Index: 300

Text #16

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.6301. Min Act: -0.0000

Data Index: 1522464 (Python Code)

Max Activating Token Index: 679

Text #17

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.5409. Min Act: -0.0000

Data Index: 1474640 (Python Code)

Max Activating Token Index: 819

Text #18

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.5386. Min Act: -0.0000

Data Index: 1446514 (Python Code)

Max Activating Token Index: 221

Text #19

Max Range: 2.9350. Min Range: -2.9350

Max Act: 1.6229. Min Act: -0.0000

Data Index: 1650766 (Python Code)

Max Activating Token Index: 745