Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4357 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.5538. Min Act: -0.0000

Data Index: 597136 (C4 (Web Text))

Max Activating Token Index: 100

Text #1

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.3662. Min Act: -0.0000

Data Index: 781317 (C4 (Web Text))

Max Activating Token Index: 822

Text #2

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.3327. Min Act: -0.0000

Data Index: 394240 (C4 (Web Text))

Max Activating Token Index: 88

Text #3

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.2167. Min Act: -0.0000

Data Index: 441360 (C4 (Web Text))

Max Activating Token Index: 65

Text #4

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.2374. Min Act: -0.0000

Data Index: 85533 (C4 (Web Text))

Max Activating Token Index: 209

Text #5

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.1183. Min Act: -0.0000

Data Index: 259240 (C4 (Web Text))

Max Activating Token Index: 456

Text #6

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.1412. Min Act: -0.0000

Data Index: 155865 (C4 (Web Text))

Max Activating Token Index: 870

Text #7

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.1021. Min Act: -0.0000

Data Index: 810719 (C4 (Web Text))

Max Activating Token Index: 223

Text #8

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0847. Min Act: -0.0000

Data Index: 388682 (C4 (Web Text))

Max Activating Token Index: 108

Text #9

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.1471. Min Act: -0.0000

Data Index: 959661 (C4 (Web Text))

Max Activating Token Index: 696

Text #10

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.1354. Min Act: -0.0000

Data Index: 1188955 (C4 (Web Text))

Max Activating Token Index: 401

Text #11

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0301. Min Act: -0.0000

Data Index: 868670 (C4 (Web Text))

Max Activating Token Index: 858

Text #12

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.1187. Min Act: -0.0000

Data Index: 167979 (C4 (Web Text))

Max Activating Token Index: 448

Text #13

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0851. Min Act: -0.0000

Data Index: 551569 (C4 (Web Text))

Max Activating Token Index: 469

Text #14

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0864. Min Act: -0.0000

Data Index: 279062 (C4 (Web Text))

Max Activating Token Index: 550

Text #15

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0813. Min Act: -0.0000

Data Index: 28288 (C4 (Web Text))

Max Activating Token Index: 83

Text #16

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0701. Min Act: -0.0000

Data Index: 303855 (C4 (Web Text))

Max Activating Token Index: 656

Text #17

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0133. Min Act: -0.0000

Data Index: 994101 (C4 (Web Text))

Max Activating Token Index: 747

Text #18

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0434. Min Act: -0.0000

Data Index: 987693 (C4 (Web Text))

Max Activating Token Index: 450

Text #19

Max Range: 1.5538. Min Range: -1.5538

Max Act: 1.0076. Min Act: -0.0000

Data Index: 1058633 (C4 (Web Text))

Max Activating Token Index: 371