Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1883 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.8340. Min Range: -3.8340

Max Act: 3.8340. Min Act: -0.0000

Data Index: 504033 (C4 (Web Text))

Max Activating Token Index: 81

Text #1

Max Range: 3.8340. Min Range: -3.8340

Max Act: 3.5445. Min Act: -0.0000

Data Index: 1292200 (C4 (Web Text))

Max Activating Token Index: 113

Text #2

Max Range: 3.8340. Min Range: -3.8340

Max Act: 3.0310. Min Act: -0.0000

Data Index: 1346653 (C4 (Web Text))

Max Activating Token Index: 430

Text #3

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.9753. Min Act: -0.0000

Data Index: 1348515 (C4 (Web Text))

Max Activating Token Index: 790

Text #4

Max Range: 3.8340. Min Range: -3.8340

Max Act: 3.0091. Min Act: -0.0000

Data Index: 217166 (C4 (Web Text))

Max Activating Token Index: 806

Text #5

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.9597. Min Act: -0.0000

Data Index: 724536 (C4 (Web Text))

Max Activating Token Index: 733

Text #6

Max Range: 3.8340. Min Range: -3.8340

Max Act: 3.1647. Min Act: -0.0000

Data Index: 483337 (C4 (Web Text))

Max Activating Token Index: 228

Text #7

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.7634. Min Act: -0.0000

Data Index: 285956 (C4 (Web Text))

Max Activating Token Index: 680

Text #8

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.8609. Min Act: -0.0000

Data Index: 328274 (C4 (Web Text))

Max Activating Token Index: 976

Text #9

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.7199. Min Act: -0.0000

Data Index: 922563 (C4 (Web Text))

Max Activating Token Index: 880

Text #10

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.8635. Min Act: -0.0000

Data Index: 417845 (C4 (Web Text))

Max Activating Token Index: 1007

Text #11

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.7017. Min Act: -0.0000

Data Index: 1134460 (C4 (Web Text))

Max Activating Token Index: 909

Text #12

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.8654. Min Act: -0.0000

Data Index: 439665 (C4 (Web Text))

Max Activating Token Index: 708

Text #13

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.8571. Min Act: -0.0000

Data Index: 771201 (C4 (Web Text))

Max Activating Token Index: 775

Text #14

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.8483. Min Act: -0.0000

Data Index: 229337 (C4 (Web Text))

Max Activating Token Index: 852

Text #15

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.6589. Min Act: -0.0000

Data Index: 1339330 (C4 (Web Text))

Max Activating Token Index: 721

Text #16

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.7418. Min Act: -0.0000

Data Index: 122359 (C4 (Web Text))

Max Activating Token Index: 422

Text #17

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.7283. Min Act: -0.0000

Data Index: 514106 (C4 (Web Text))

Max Activating Token Index: 506

Text #18

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.7327. Min Act: -0.0000

Data Index: 1336558 (C4 (Web Text))

Max Activating Token Index: 350

Text #19

Max Range: 3.8340. Min Range: -3.8340

Max Act: 2.6134. Min Act: -0.0000

Data Index: 555964 (C4 (Web Text))

Max Activating Token Index: 502