Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 102 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.7277. Min Act: -0.0000

Data Index: 148643 (The Pile)

Max Activating Token Index: 1009

Text #1

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.6571. Min Act: -0.0000

Data Index: 303353 (The Pile)

Max Activating Token Index: 520

Text #2

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.8339. Min Act: -0.0000

Data Index: 1811028 (The Pile)

Max Activating Token Index: 276

Text #3

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.6608. Min Act: -0.0000

Data Index: 1570172 (The Pile)

Max Activating Token Index: 161

Text #4

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.0875. Min Act: -0.0000

Data Index: 798410 (The Pile)

Max Activating Token Index: 164

Text #5

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.2134. Min Act: -0.0000

Data Index: 315384 (The Pile)

Max Activating Token Index: 136

Text #6

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.6308. Min Act: -0.0000

Data Index: 1698656 (The Pile)

Max Activating Token Index: 691

Text #7

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.1870. Min Act: -0.0000

Data Index: 1825158 (The Pile)

Max Activating Token Index: 530

Text #8

Max Range: 3.8339. Min Range: -3.8339

Max Act: 2.8394. Min Act: -0.0000

Data Index: 441379 (The Pile)

Max Activating Token Index: 584

Text #9

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.1103. Min Act: -0.0000

Data Index: 1646859 (The Pile)

Max Activating Token Index: 441

Text #10

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.1057. Min Act: -0.0000

Data Index: 1564413 (The Pile)

Max Activating Token Index: 965

Text #11

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.2342. Min Act: -0.0000

Data Index: 1874313 (The Pile)

Max Activating Token Index: 732

Text #12

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.3431. Min Act: -0.0000

Data Index: 499751 (The Pile)

Max Activating Token Index: 806

Text #13

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.0036. Min Act: -0.0000

Data Index: 171381 (The Pile)

Max Activating Token Index: 365

Text #14

Max Range: 3.8339. Min Range: -3.8339

Max Act: 2.8624. Min Act: -0.0000

Data Index: 341820 (The Pile)

Max Activating Token Index: 684

Text #15

Max Range: 3.8339. Min Range: -3.8339

Max Act: 2.8595. Min Act: -0.0000

Data Index: 315541 (The Pile)

Max Activating Token Index: 230

Text #16

Max Range: 3.8339. Min Range: -3.8339

Max Act: 3.0037. Min Act: -0.0000

Data Index: 996099 (The Pile)

Max Activating Token Index: 248

Text #17

Max Range: 3.8339. Min Range: -3.8339

Max Act: 2.9800. Min Act: -0.0000

Data Index: 319482 (The Pile)

Max Activating Token Index: 350

Text #18

Max Range: 3.8339. Min Range: -3.8339

Max Act: 2.9793. Min Act: -0.0000

Data Index: 1891421 (The Pile)

Max Activating Token Index: 458

Text #19

Max Range: 3.8339. Min Range: -3.8339

Max Act: 2.9708. Min Act: -0.0000

Data Index: 583054 (The Pile)

Max Activating Token Index: 629