Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 375 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.3862. Min Range: -3.3862

Max Act: 3.3862. Min Act: -0.0000

Data Index: 1881712 (The Pile)

Max Activating Token Index: 879

Text #1

Max Range: 3.3862. Min Range: -3.3862

Max Act: 3.2023. Min Act: -0.0000

Data Index: 1811728 (The Pile)

Max Activating Token Index: 561

Text #2

Max Range: 3.3862. Min Range: -3.3862

Max Act: 3.3099. Min Act: -0.0001

Data Index: 1562252 (The Pile)

Max Activating Token Index: 769

Text #3

Max Range: 3.3862. Min Range: -3.3862

Max Act: 3.0238. Min Act: -0.0000

Data Index: 223857 (The Pile)

Max Activating Token Index: 15

Text #4

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.9749. Min Act: -0.0000

Data Index: 836631 (The Pile)

Max Activating Token Index: 634

Text #5

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.7814. Min Act: -0.0000

Data Index: 1710084 (The Pile)

Max Activating Token Index: 41

Text #6

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.8803. Min Act: -0.0000

Data Index: 193795 (The Pile)

Max Activating Token Index: 425

Text #7

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.7337. Min Act: -0.0000

Data Index: 952770 (The Pile)

Max Activating Token Index: 198

Text #8

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.9809. Min Act: -0.0000

Data Index: 1536682 (The Pile)

Max Activating Token Index: 576

Text #9

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.8547. Min Act: -0.0000

Data Index: 1993724 (The Pile)

Max Activating Token Index: 1002

Text #10

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.8040. Min Act: -0.0000

Data Index: 1850473 (The Pile)

Max Activating Token Index: 234

Text #11

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.6234. Min Act: -0.0000

Data Index: 1137040 (The Pile)

Max Activating Token Index: 380

Text #12

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.6065. Min Act: -0.0000

Data Index: 454942 (The Pile)

Max Activating Token Index: 771

Text #13

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.8521. Min Act: -0.0000

Data Index: 1543975 (The Pile)

Max Activating Token Index: 496

Text #14

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.5295. Min Act: -0.0000

Data Index: 1894433 (The Pile)

Max Activating Token Index: 659

Text #15

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.6921. Min Act: -0.0000

Data Index: 177646 (The Pile)

Max Activating Token Index: 136

Text #16

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.7845. Min Act: -0.0000

Data Index: 232920 (The Pile)

Max Activating Token Index: 288

Text #17

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.6416. Min Act: -0.0000

Data Index: 762414 (The Pile)

Max Activating Token Index: 288

Text #18

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.6247. Min Act: -0.0000

Data Index: 1270663 (The Pile)

Max Activating Token Index: 964

Text #19

Max Range: 3.3862. Min Range: -3.3862

Max Act: 2.4912. Min Act: -0.0000

Data Index: 1611296 (The Pile)

Max Activating Token Index: 683