Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3486 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.8489. Min Act: -0.0000

Data Index: 330264 (C4 (Web Text))

Max Activating Token Index: 752

Text #1

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.6603. Min Act: -0.0000

Data Index: 762268 (C4 (Web Text))

Max Activating Token Index: 883

Text #2

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.5933. Min Act: -0.0000

Data Index: 24162 (C4 (Web Text))

Max Activating Token Index: 553

Text #3

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.5009. Min Act: -0.0000

Data Index: 709978 (C4 (Web Text))

Max Activating Token Index: 457

Text #4

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.6446. Min Act: -0.0000

Data Index: 229504 (C4 (Web Text))

Max Activating Token Index: 381

Text #5

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.5689. Min Act: -0.0000

Data Index: 286469 (C4 (Web Text))

Max Activating Token Index: 115

Text #6

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.3886. Min Act: -0.0000

Data Index: 1229384 (C4 (Web Text))

Max Activating Token Index: 54

Text #7

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.3612. Min Act: -0.0000

Data Index: 529030 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.2856. Min Act: -0.0000

Data Index: 269194 (C4 (Web Text))

Max Activating Token Index: 783

Text #9

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.2677. Min Act: -0.0000

Data Index: 910418 (C4 (Web Text))

Max Activating Token Index: 95

Text #10

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.3210. Min Act: -0.0000

Data Index: 389204 (C4 (Web Text))

Max Activating Token Index: 438

Text #11

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1826. Min Act: -0.0000

Data Index: 833562 (C4 (Web Text))

Max Activating Token Index: 646

Text #12

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1605. Min Act: -0.0000

Data Index: 656113 (C4 (Web Text))

Max Activating Token Index: 446

Text #13

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1324. Min Act: -0.0000

Data Index: 1282525 (C4 (Web Text))

Max Activating Token Index: 861

Text #14

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1714. Min Act: -0.0000

Data Index: 229594 (C4 (Web Text))

Max Activating Token Index: 458

Text #15

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.0995. Min Act: -0.0000

Data Index: 1293545 (C4 (Web Text))

Max Activating Token Index: 265

Text #16

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1456. Min Act: -0.0000

Data Index: 1348191 (C4 (Web Text))

Max Activating Token Index: 941

Text #17

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1412. Min Act: -0.0000

Data Index: 34182 (C4 (Web Text))

Max Activating Token Index: 825

Text #18

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.1831. Min Act: -0.0000

Data Index: 1138245 (C4 (Web Text))

Max Activating Token Index: 278

Text #19

Max Range: 1.8489. Min Range: -1.8489

Max Act: 1.0775. Min Act: -0.0000

Data Index: 113279 (C4 (Web Text))

Max Activating Token Index: 807