Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4669 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.7541. Min Act: -0.0000

Data Index: 712813 (The Pile)

Max Activating Token Index: 108

Text #1

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.4948. Min Act: -0.0000

Data Index: 856529 (The Pile)

Max Activating Token Index: 566

Text #2

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.3824. Min Act: -0.0000

Data Index: 1860962 (The Pile)

Max Activating Token Index: 95

Text #3

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.0258. Min Act: -0.0000

Data Index: 1329208 (The Pile)

Max Activating Token Index: 165

Text #4

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.1927. Min Act: -0.0000

Data Index: 1369597 (The Pile)

Max Activating Token Index: 33

Text #5

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.1032. Min Act: -0.0000

Data Index: 1114012 (The Pile)

Max Activating Token Index: 280

Text #6

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.0559. Min Act: -0.0000

Data Index: 282301 (The Pile)

Max Activating Token Index: 132

Text #7

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.8493. Min Act: -0.0000

Data Index: 1493986 (The Pile)

Max Activating Token Index: 335

Text #8

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.4658. Min Act: -0.0000

Data Index: 1072891 (The Pile)

Max Activating Token Index: 751

Text #9

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.4331. Min Act: -0.0000

Data Index: 1069825 (The Pile)

Max Activating Token Index: 79

Text #10

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.6141. Min Act: -0.0000

Data Index: 1199815 (The Pile)

Max Activating Token Index: 46

Text #11

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.7762. Min Act: -0.0000

Data Index: 992602 (The Pile)

Max Activating Token Index: 194

Text #12

Max Range: 5.7541. Min Range: -5.7541

Max Act: 5.0999. Min Act: -0.0000

Data Index: 1694904 (The Pile)

Max Activating Token Index: 227

Text #13

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.7549. Min Act: -0.0000

Data Index: 427900 (The Pile)

Max Activating Token Index: 443

Text #14

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.5547. Min Act: -0.0000

Data Index: 1124270 (The Pile)

Max Activating Token Index: 49

Text #15

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.9078. Min Act: -0.0000

Data Index: 1672336 (The Pile)

Max Activating Token Index: 337

Text #16

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.7119. Min Act: -0.0000

Data Index: 888981 (The Pile)

Max Activating Token Index: 304

Text #17

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.3689. Min Act: -0.0000

Data Index: 142898 (The Pile)

Max Activating Token Index: 451

Text #18

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.5060. Min Act: -0.0000

Data Index: 970187 (The Pile)

Max Activating Token Index: 75

Text #19

Max Range: 5.7541. Min Range: -5.7541

Max Act: 4.5102. Min Act: -0.0000

Data Index: 353997 (The Pile)

Max Activating Token Index: 265