Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3353 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.5320. Min Range: -3.5320

Max Act: 3.5320. Min Act: -0.0000

Data Index: 431253 (C4 (Web Text))

Max Activating Token Index: 685

Text #1

Max Range: 3.5320. Min Range: -3.5320

Max Act: 3.2605. Min Act: -0.0000

Data Index: 1033346 (C4 (Web Text))

Max Activating Token Index: 676

Text #2

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.7225. Min Act: -0.0000

Data Index: 649822 (C4 (Web Text))

Max Activating Token Index: 674

Text #3

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.6609. Min Act: -0.0000

Data Index: 673868 (C4 (Web Text))

Max Activating Token Index: 46

Text #4

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.4400. Min Act: -0.0000

Data Index: 836981 (C4 (Web Text))

Max Activating Token Index: 423

Text #5

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.5070. Min Act: -0.0000

Data Index: 436687 (C4 (Web Text))

Max Activating Token Index: 203

Text #6

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.1631. Min Act: -0.0000

Data Index: 147356 (C4 (Web Text))

Max Activating Token Index: 417

Text #7

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.0597. Min Act: -0.0000

Data Index: 1083951 (C4 (Web Text))

Max Activating Token Index: 539

Text #8

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.1904. Min Act: -0.0000

Data Index: 1157806 (C4 (Web Text))

Max Activating Token Index: 642

Text #9

Max Range: 3.5320. Min Range: -3.5320

Max Act: 2.0350. Min Act: -0.0000

Data Index: 1240409 (C4 (Web Text))

Max Activating Token Index: 684

Text #10

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.9296. Min Act: -0.0000

Data Index: 321802 (C4 (Web Text))

Max Activating Token Index: 450

Text #11

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.9027. Min Act: -0.0000

Data Index: 90081 (C4 (Web Text))

Max Activating Token Index: 364

Text #12

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.8382. Min Act: -0.0000

Data Index: 779533 (C4 (Web Text))

Max Activating Token Index: 442

Text #13

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.9581. Min Act: -0.0000

Data Index: 805492 (C4 (Web Text))

Max Activating Token Index: 946

Text #14

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.7562. Min Act: -0.0000

Data Index: 1334765 (C4 (Web Text))

Max Activating Token Index: 900

Text #15

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.9614. Min Act: -0.0000

Data Index: 731548 (C4 (Web Text))

Max Activating Token Index: 215

Text #16

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.6323. Min Act: -0.0000

Data Index: 1290663 (C4 (Web Text))

Max Activating Token Index: 326

Text #17

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.5710. Min Act: -0.0000

Data Index: 1201793 (C4 (Web Text))

Max Activating Token Index: 260

Text #18

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.9001. Min Act: -0.0000

Data Index: 297845 (C4 (Web Text))

Max Activating Token Index: 72

Text #19

Max Range: 3.5320. Min Range: -3.5320

Max Act: 1.9151. Min Act: -0.0000

Data Index: 31342 (C4 (Web Text))

Max Activating Token Index: 197