Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1978 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.0433. Min Range: -3.0433

Max Act: 3.0433. Min Act: -0.0000

Data Index: 1835027 (The Pile)

Max Activating Token Index: 659

Text #1

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.7224. Min Act: -0.0000

Data Index: 1618290 (The Pile)

Max Activating Token Index: 404

Text #2

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.6196. Min Act: -0.0000

Data Index: 1695434 (The Pile)

Max Activating Token Index: 911

Text #3

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.8696. Min Act: -0.0000

Data Index: 283221 (The Pile)

Max Activating Token Index: 290

Text #4

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.7006. Min Act: -0.0000

Data Index: 1702862 (The Pile)

Max Activating Token Index: 689

Text #5

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.5168. Min Act: -0.0000

Data Index: 1026636 (The Pile)

Max Activating Token Index: 1015

Text #6

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.4551. Min Act: -0.0000

Data Index: 418893 (The Pile)

Max Activating Token Index: 172

Text #7

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.5268. Min Act: -0.0000

Data Index: 444044 (The Pile)

Max Activating Token Index: 954

Text #8

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.5865. Min Act: -0.0000

Data Index: 84351 (The Pile)

Max Activating Token Index: 655

Text #9

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.5082. Min Act: -0.0000

Data Index: 999863 (The Pile)

Max Activating Token Index: 954

Text #10

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.5343. Min Act: -0.0000

Data Index: 962565 (The Pile)

Max Activating Token Index: 546

Text #11

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.4800. Min Act: -0.0000

Data Index: 1110474 (The Pile)

Max Activating Token Index: 793

Text #12

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.3859. Min Act: -0.0000

Data Index: 1280881 (The Pile)

Max Activating Token Index: 90

Text #13

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.2306. Min Act: -0.0000

Data Index: 566868 (The Pile)

Max Activating Token Index: 645

Text #14

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.3576. Min Act: -0.0000

Data Index: 314242 (The Pile)

Max Activating Token Index: 661

Text #15

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.4804. Min Act: -0.0000

Data Index: 1432176 (The Pile)

Max Activating Token Index: 944

Text #16

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.4238. Min Act: -0.0000

Data Index: 1748514 (The Pile)

Max Activating Token Index: 220

Text #17

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.4263. Min Act: -0.0000

Data Index: 997891 (The Pile)

Max Activating Token Index: 489

Text #18

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.5069. Min Act: -0.0000

Data Index: 222555 (The Pile)

Max Activating Token Index: 434

Text #19

Max Range: 3.0433. Min Range: -3.0433

Max Act: 1.3280. Min Act: -0.0000

Data Index: 918868 (The Pile)

Max Activating Token Index: 735