Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3805 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.7288. Min Act: -0.0000

Data Index: 1087516 (The Pile)

Max Activating Token Index: 513

Text #1

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.4928. Min Act: -0.0000

Data Index: 1372461 (The Pile)

Max Activating Token Index: 440

Text #2

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.5607. Min Act: -0.0000

Data Index: 163987 (The Pile)

Max Activating Token Index: 184

Text #3

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.3049. Min Act: -0.0000

Data Index: 672189 (The Pile)

Max Activating Token Index: 159

Text #4

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.3636. Min Act: -0.0000

Data Index: 272070 (The Pile)

Max Activating Token Index: 900

Text #5

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.3419. Min Act: -0.0000

Data Index: 1832936 (The Pile)

Max Activating Token Index: 357

Text #6

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.2390. Min Act: -0.0000

Data Index: 923757 (The Pile)

Max Activating Token Index: 405

Text #7

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.2276. Min Act: -0.0000

Data Index: 172861 (The Pile)

Max Activating Token Index: 338

Text #8

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.2750. Min Act: -0.0000

Data Index: 1609557 (The Pile)

Max Activating Token Index: 148

Text #9

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1869. Min Act: -0.0000

Data Index: 1483282 (The Pile)

Max Activating Token Index: 964

Text #10

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1761. Min Act: -0.0000

Data Index: 1738365 (The Pile)

Max Activating Token Index: 771

Text #11

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1428. Min Act: -0.0000

Data Index: 653645 (The Pile)

Max Activating Token Index: 446

Text #12

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1350. Min Act: -0.0000

Data Index: 1174384 (The Pile)

Max Activating Token Index: 746

Text #13

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1833. Min Act: -0.0000

Data Index: 644482 (The Pile)

Max Activating Token Index: 133

Text #14

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1659. Min Act: -0.0000

Data Index: 310865 (The Pile)

Max Activating Token Index: 1014

Text #15

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.2256. Min Act: -0.0000

Data Index: 1900036 (The Pile)

Max Activating Token Index: 307

Text #16

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1391. Min Act: -0.0000

Data Index: 887275 (The Pile)

Max Activating Token Index: 920

Text #17

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.2209. Min Act: -0.0000

Data Index: 1607291 (The Pile)

Max Activating Token Index: 192

Text #18

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.1148. Min Act: -0.0000

Data Index: 741366 (The Pile)

Max Activating Token Index: 751

Text #19

Max Range: 1.7288. Min Range: -1.7288

Max Act: 1.2490. Min Act: -0.0000

Data Index: 1612269 (The Pile)

Max Activating Token Index: 713