Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3638 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.5268. Min Act: -0.0000

Data Index: 1124005 (The Pile)

Max Activating Token Index: 962

Text #1

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.5217. Min Act: -0.0000

Data Index: 1526243 (The Pile)

Max Activating Token Index: 443

Text #2

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.5442. Min Act: -0.0000

Data Index: 1423037 (The Pile)

Max Activating Token Index: 797

Text #3

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.3964. Min Act: -0.0000

Data Index: 649876 (The Pile)

Max Activating Token Index: 841

Text #4

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.3382. Min Act: -0.0000

Data Index: 420113 (The Pile)

Max Activating Token Index: 563

Text #5

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.3222. Min Act: -0.0000

Data Index: 1128203 (The Pile)

Max Activating Token Index: 163

Text #6

Max Range: 3.5442. Min Range: -3.5442

Max Act: 2.8583. Min Act: -0.0000

Data Index: 1230936 (The Pile)

Max Activating Token Index: 469

Text #7

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1515. Min Act: -0.0000

Data Index: 8162 (The Pile)

Max Activating Token Index: 541

Text #8

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1389. Min Act: -0.0000

Data Index: 905046 (The Pile)

Max Activating Token Index: 678

Text #9

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.2587. Min Act: -0.0000

Data Index: 911881 (The Pile)

Max Activating Token Index: 464

Text #10

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1790. Min Act: -0.0000

Data Index: 469201 (The Pile)

Max Activating Token Index: 696

Text #11

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.2235. Min Act: -0.0000

Data Index: 1461228 (The Pile)

Max Activating Token Index: 590

Text #12

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.0596. Min Act: -0.0000

Data Index: 509794 (The Pile)

Max Activating Token Index: 235

Text #13

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1928. Min Act: -0.0000

Data Index: 849381 (The Pile)

Max Activating Token Index: 515

Text #14

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1932. Min Act: -0.0000

Data Index: 1793976 (The Pile)

Max Activating Token Index: 736

Text #15

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.0529. Min Act: -0.0000

Data Index: 1217956 (The Pile)

Max Activating Token Index: 630

Text #16

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.0464. Min Act: -0.0000

Data Index: 683754 (The Pile)

Max Activating Token Index: 424

Text #17

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1720. Min Act: -0.0000

Data Index: 462730 (The Pile)

Max Activating Token Index: 170

Text #18

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1586. Min Act: -0.0000

Data Index: 916923 (The Pile)

Max Activating Token Index: 373

Text #19

Max Range: 3.5442. Min Range: -3.5442

Max Act: 3.1649. Min Act: -0.0000

Data Index: 933713 (The Pile)

Max Activating Token Index: 734