Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2928 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.9531. Min Act: -0.0000

Data Index: 1070328 (C4 (Web Text))

Max Activating Token Index: 695

Text #1

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.8561. Min Act: -0.0000

Data Index: 1357685 (C4 (Web Text))

Max Activating Token Index: 561

Text #2

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.8020. Min Act: -0.0000

Data Index: 526304 (C4 (Web Text))

Max Activating Token Index: 613

Text #3

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.7213. Min Act: -0.0000

Data Index: 1218024 (C4 (Web Text))

Max Activating Token Index: 642

Text #4

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.3404. Min Act: -0.0000

Data Index: 1088743 (C4 (Web Text))

Max Activating Token Index: 612

Text #5

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.4216. Min Act: -0.0000

Data Index: 1354984 (C4 (Web Text))

Max Activating Token Index: 372

Text #6

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.5562. Min Act: -0.0000

Data Index: 719805 (C4 (Web Text))

Max Activating Token Index: 456

Text #7

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.6995. Min Act: -0.0000

Data Index: 483532 (C4 (Web Text))

Max Activating Token Index: 557

Text #8

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.4215. Min Act: -0.0000

Data Index: 1243977 (C4 (Web Text))

Max Activating Token Index: 468

Text #9

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.5344. Min Act: -0.0000

Data Index: 1135813 (C4 (Web Text))

Max Activating Token Index: 685

Text #10

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.3221. Min Act: -0.0000

Data Index: 237499 (C4 (Web Text))

Max Activating Token Index: 597

Text #11

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.6738. Min Act: -0.0000

Data Index: 1335679 (C4 (Web Text))

Max Activating Token Index: 583

Text #12

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.4971. Min Act: -0.0000

Data Index: 658604 (C4 (Web Text))

Max Activating Token Index: 823

Text #13

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.3722. Min Act: -0.0000

Data Index: 488139 (C4 (Web Text))

Max Activating Token Index: 570

Text #14

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.1320. Min Act: -0.0000

Data Index: 1124457 (C4 (Web Text))

Max Activating Token Index: 189

Text #15

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.0867. Min Act: -0.0000

Data Index: 778038 (C4 (Web Text))

Max Activating Token Index: 451

Text #16

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.0458. Min Act: -0.0000

Data Index: 773437 (C4 (Web Text))

Max Activating Token Index: 509

Text #17

Max Range: 3.9531. Min Range: -3.9531

Max Act: 2.8756. Min Act: -0.0000

Data Index: 567066 (C4 (Web Text))

Max Activating Token Index: 508

Text #18

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.3287. Min Act: -0.0000

Data Index: 66383 (C4 (Web Text))

Max Activating Token Index: 469

Text #19

Max Range: 3.9531. Min Range: -3.9531

Max Act: 3.3156. Min Act: -0.0000

Data Index: 304402 (C4 (Web Text))

Max Activating Token Index: 939