Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 382 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0415. Min Act: -0.0000

Data Index: 473360 (C4 (Web Text))

Max Activating Token Index: 696

Text #1

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0411. Min Act: -0.0000

Data Index: 1556357 (Python Code)

Max Activating Token Index: 988

Text #2

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0404. Min Act: -0.0000

Data Index: 564448 (C4 (Web Text))

Max Activating Token Index: 226

Text #3

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0392. Min Act: -0.0000

Data Index: 224052 (C4 (Web Text))

Max Activating Token Index: 598

Text #4

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0409. Min Act: -0.0000

Data Index: 512850 (C4 (Web Text))

Max Activating Token Index: 95

Text #5

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0394. Min Act: -0.0000

Data Index: 297287 (C4 (Web Text))

Max Activating Token Index: 626

Text #6

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0396. Min Act: -0.0000

Data Index: 656011 (C4 (Web Text))

Max Activating Token Index: 488

Text #7

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0390. Min Act: -0.0000

Data Index: 347622 (C4 (Web Text))

Max Activating Token Index: 412

Text #8

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0346. Min Act: -0.0000

Data Index: 831099 (C4 (Web Text))

Max Activating Token Index: 508

Text #9

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0332. Min Act: -0.0000

Data Index: 4032 (C4 (Web Text))

Max Activating Token Index: 920

Text #10

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0358. Min Act: -0.0000

Data Index: 1549653 (Python Code)

Max Activating Token Index: 1002

Text #11

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0344. Min Act: -0.0000

Data Index: 583812 (C4 (Web Text))

Max Activating Token Index: 63

Text #12

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0334. Min Act: -0.0000

Data Index: 1627698 (Python Code)

Max Activating Token Index: 990

Text #13

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0338. Min Act: -0.0000

Data Index: 920619 (C4 (Web Text))

Max Activating Token Index: 684

Text #14

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0327. Min Act: -0.0000

Data Index: 96166 (C4 (Web Text))

Max Activating Token Index: 863

Text #15

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0356. Min Act: -0.0000

Data Index: 461091 (C4 (Web Text))

Max Activating Token Index: 831

Text #16

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0336. Min Act: -0.0000

Data Index: 108914 (C4 (Web Text))

Max Activating Token Index: 74

Text #17

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0322. Min Act: -0.0000

Data Index: 985523 (C4 (Web Text))

Max Activating Token Index: 413

Text #18

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0347. Min Act: -0.0000

Data Index: 465148 (C4 (Web Text))

Max Activating Token Index: 938

Text #19

Max Range: 0.0415. Min Range: -0.0415

Max Act: 0.0301. Min Act: -0.0000

Data Index: 1573588 (Python Code)

Max Activating Token Index: 827